Now that we've got that out of the way, I wouldn't be blogging about this if it hadn't worked.
A little bit of background: I troubleshot this error back in 2012, so I'm working from my notes. I believe the 'unexpected-error' showed up in the run history, but I can't remember if FIM Sync actually crashed; maybe this will help someone out there anyway. At the time, I recall troubleshooting an actual crash of miiserver.exe, so unfortunately I don't remember if we had tried storechk.exe before I fixed the database with this hack. Anyway, here goes...
I've pasted the event log entry at the bottom of this post for reference. General steps to fix are:
1. Enumerate orphaned CS-MV links
2. Save off orphaned links
3. Delete CS records
4. Delete links
5. Full import from affected MAs
6. Full syncs as necessary
1. Enumerate orphaned CS-MV links
2. Save off orphaned links
3. Delete CS records
4. Delete links
5. Full import from affected MAs
6. Full syncs as necessary
1. Enumerate orphaned CS-MV links
select cs.ma_id, ma.[ma_name] ,count(*) as [count],min([initial_import_error_date]) as [min initial import error date]
from dbo.mms_connectorspace cs
join [dbo].[mms_management_agent] ma
on ma.[ma_id] = cs.[ma_id]
join (
SELECT [mv_object_id]
,mv.[object_id] as [mv.object_id]
,[cs_object_id]
,[lineage_id]
,[lineage_date]
FROM [FIMSynchronizationService].[dbo].[mms_csmv_link] csmv
full outer join [FIMSynchronizationService].[dbo].[mms_metaverse] mv
on mv.[object_id] = csmv.[mv_object_id]
where mv.[object_id] is null
)b
on b.cs_object_id = cs.object_id
group by cs.ma_id, ma.[ma_name]
order by count(*) desc
ma_name
|
count
|
min initial import error date
|
SQL - Roles
|
764
|
10/17/12 6:43 AM
|
SQL - Virtual
|
348
|
NULL
|
SQL - 3rd party
|
277
|
10/16/12 8:16 PM
|
SQL - Principal
|
277
|
10/16/12 8:09 PM
|
AD-LDS - Partners
|
251
|
10/16/12 9:06 PM
|
SQL - Groups
|
238
|
10/16/12 8:07 PM
|
SQL - SSO
|
238
|
10/16/12 8:08 PM
|
SQL - Mapping
|
2
|
10/17/12 5:27 AM
|
SQL - User gen
|
2
|
10/17/12 5:43 AM
|
AD - Primary
|
2
|
10/17/12 3:51 AM
|
ECMA - Java
|
1
|
10/17/12 3:40 AM
|
ECMA - Web services
|
1
|
10/17/12 3:47 AM
|
2. Save off orphaned links
SELECT [mv_object_id]
,mv.[object_id] as [mv.object_id]
,[cs_object_id]
,[lineage_id]
,[lineage_date]
into #wehackedit
FROM [FIMSynchronizationService].[dbo].[mms_csmv_link] csmv
full outer join [FIMSynchronizationService].[dbo].[mms_metaverse] mv
on mv.[object_id] = csmv.[mv_object_id]
where mv.[object_id] is null
3. Delete CS records
delete from dbo.mms_connectorspace
where object_id in
(
select cs_object_id from #wehackedit
)
4. Delete links
delete from dbo.mms_csmv_link
where mv_object_id in
(
select [mv_object_id] from #wehackedit
)
Event log
Event Type: Error
Event Source: MIIServer
Event Category: Server
Event ID: 6301
Date: 10/18/2012
Time: 12:17:53 AM
User: N/A
Computer: FIMSYNC01
Description:
The server encountered an unexpected error in the synchronization engine:
"BAIL: MMS(7924): sproc.cpp(1198): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): imgbldr.cpp(1216): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): imgbldr.cpp(1078): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): imgbldr.cpp(3145): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): mvobj.cpp(199): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): nsmvimp.cpp(274): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): csobj.cpp(2062): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): synccore.cpp(555): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): synccoreimp.cpp(118): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): synccoreimp.cpp(5842): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): synccoreimp.cpp(2224): 0x80230405 (The operation failed because the object cannot be found)
ERR: MMS(7924): synccoreimp.cpp(2240): 0x80230405 - CS to MV to CS synchronization failed 0x80230405: [U12345678]
BAIL: MMS(7924): synccoreimp.cpp(2076): 0x80230405 (The operation failed because the object cannot be found)
ERR: MMS(7924): syncmonitor.cpp(2502): SE: Rollback SQL transaction for: 0x80230405
MMS(7924): SE: CS image begin
MMS(7924): <cs-object cs-dn="U12345678" id="{3FDD7506-5A93-4572-A60C-D22F4F46BEFC}" object-type="person">
<unapplied-export>
<delta operation="none" dn="U12345678">
<anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
</delta>
</unapplied-export>
<escrowed-export>
<delta operation="none" dn="U12345678">
<anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
</delta>
</escrowed-export>
<unconfirmed-export>
<delta operation="none" dn="U12345678">
<anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
</delta>
</unconfirmed-export>
<pending-import>
<delta operation="none" dn="U12345678">
<anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
</delta>
</pending-import>
<synchronized-hologram>
<entry dn="U12345678">
<anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
<primary-objectclass>person</primary-objectclass>
<objectclass>
<oc-value>person</oc-value>
</objectclass>
<attr name="DepartmentId" type="string" multivalued="false">
<value>4321</value>
</attr>
<attr name="EmployeeId" type="string" multivalued="false">
<value>U12345678</value>
</attr>
<attr name="UserId" type="string" multivalued="false">
<value>U12345678</value>
</attr>
</entry>
</synchronized-hologram>
<anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
<connector>1</connector>
<connector-state>normal</connector-state>
<seen-by-import>1</seen-by-import>
<rebuild-in-progress>0</rebuild-in-progress>
<obsoletion>0</obsoletion>
<need-full-sync>0</need-full-sync>
<placeholder-parent>0</placeholder-parent>
<placeholder-link>0</placeholder-link>
<placeholder-delete>0</placeholder-delete>
<pending>0</pending>
<ref-retry>0</ref-retry>
<rename-retry>0</rename-retry>
<sequencers>
<current>
<batch-number>31008</batch-number>
<sequence-number>1529595</sequence-number>
</current>
<unapplied>
<batch-number>31008</batch-number>
<sequence-number>1529595</sequence-number>
</unapplied>
<original>
<batch-number>31008</batch-number>
<sequence-number>1529595</sequence-number>
</original>
</sequencers>
<import-delta-operation>none</import-delta-operation>
<export-delta-operation>none</export-delta-operation>
<pending-ref-delete>0</pending-ref-delete>
<ma-id>{410910BB-AACB-41A5-B622-7F40F246D26C}</ma-id>
<ma-name>SQL - Roles</ma-name>
<partition-id>{BA290343-1315-40EF-96AF-32A2D0069B36}</partition-id>
<import-errordetail first-occurred="2012-10-17 06:45:38.283" date-occurred="2012-10-18 07:08:11.347" retry-count="16" error-type="unexpected-error">
<import-status>
</import-status>
</import-errordetail>
<mv-link lineage-id="{F434C74C-B9B8-4BCD-858B-B221447E1FC7}" lineage-type="provisioning-rules" lineage-time="2012-09-08 07:14:07.080">{9FC0E55D-F64C-4237-9255-8B3C9756C4CE}</mv-link>
<last-import-delta-time>2012-10-18 06:14:51.587</last-import-delta-time>
<last-export-delta-time>2012-10-16 07:36:12.437</last-export-delta-time>
</cs-object>
MMS(7924): SE: CS image end
Microsoft Identity Integration Server 3.3.0118.0"
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
