How to: fix unexpected-error (not supported)

Obligatory disclaimer: This method of fixing your FIM Sync database manually is not supported; use at your own risk! As always, back up your FIM Sync database before attempting any such operation.

Now that we've got that out of the way, I wouldn't be blogging about this if it hadn't worked.

A little bit of background: I troubleshot this error back in 2012, so I'm working from my notes. I believe the 'unexpected-error' showed up in the run history, but I can't remember if FIM Sync actually crashed; maybe this will help someone out there anyway. At the time, I recall troubleshooting an actual crash of miiserver.exe, so unfortunately I don't remember if we had tried storechk.exe before I fixed the database with this hack.  Anyway, here goes...

I've pasted the event log entry at the bottom of this post for reference.  General steps to fix are:

1. Enumerate orphaned CS-MV links
2. Save off orphaned links
3. Delete CS records
4. Delete links
5. Full import from affected MAs
6. Full syncs as necessary

1. Enumerate orphaned CS-MV links

select cs.ma_id, ma.[ma_name] ,count(*) as [count],min([initial_import_error_date]) as [min initial import error date]

from dbo.mms_connectorspace cs

join [dbo].[mms_management_agent] ma

on ma.[ma_id] = cs.[ma_id]

join (

SELECT [mv_object_id]

,mv.[object_id] as [mv.object_id]

      ,[cs_object_id]

      ,[lineage_id]

      ,[lineage_date]

  FROM [FIMSynchronizationService].[dbo].[mms_csmv_link] csmv

full outer join [FIMSynchronizationService].[dbo].[mms_metaverse] mv

on mv.[object_id] = csmv.[mv_object_id]

where mv.[object_id] is null

)b

on b.cs_object_id = cs.object_id

group by cs.ma_id, ma.[ma_name]

order by count(*) desc

ma_name	count	min initial import error date
SQL - Roles	764	10/17/12 6:43 AM
SQL - Virtual	348	NULL
SQL - 3rd party	277	10/16/12 8:16 PM
SQL - Principal	277	10/16/12 8:09 PM
AD-LDS - Partners	251	10/16/12 9:06 PM
SQL - Groups	238	10/16/12 8:07 PM
SQL - SSO	238	10/16/12 8:08 PM
SQL - Mapping	2	10/17/12 5:27 AM
SQL - User gen	2	10/17/12 5:43 AM
AD - Primary	2	10/17/12 3:51 AM
ECMA - Java	1	10/17/12 3:40 AM
ECMA - Web services	1	10/17/12 3:47 AM

2. Save off orphaned links

SELECT [mv_object_id]

,mv.[object_id] as [mv.object_id]

      ,[cs_object_id]

      ,[lineage_id]

      ,[lineage_date]

into #wehackedit

  FROM [FIMSynchronizationService].[dbo].[mms_csmv_link] csmv

full outer join [FIMSynchronizationService].[dbo].[mms_metaverse] mv

on mv.[object_id] = csmv.[mv_object_id]

where mv.[object_id] is null

3. Delete CS records

delete from dbo.mms_connectorspace

where object_id in

(

select cs_object_id from #wehackedit

)

4. Delete links

delete from dbo.mms_csmv_link

where mv_object_id in

(

select [mv_object_id] from #wehackedit

)

Event log

Event Type:    Error

Event Source:    MIIServer

Event Category:    Server

Event ID:    6301

Date:        10/18/2012

Time:        12:17:53 AM

User:        N/A

Computer:    FIMSYNC01

Description:

The server encountered an unexpected error in the synchronization engine:

"BAIL: MMS(7924): sproc.cpp(1198): 0x80230405 (The operation failed because the object cannot be found)

BAIL: MMS(7924): imgbldr.cpp(1216): 0x80230405 (The operation failed because the object cannot be found)

BAIL: MMS(7924): imgbldr.cpp(1078): 0x80230405 (The operation failed because the object cannot be found)

BAIL: MMS(7924): imgbldr.cpp(3145): 0x80230405 (The operation failed because the object cannot be found)

BAIL: MMS(7924): mvobj.cpp(199): 0x80230405 (The operation failed because the object cannot be found)

BAIL: MMS(7924): nsmvimp.cpp(274): 0x80230405 (The operation failed because the object cannot be found)

BAIL: MMS(7924): csobj.cpp(2062): 0x80230405 (The operation failed because the object cannot be found)

BAIL: MMS(7924): synccore.cpp(555): 0x80230405 (The operation failed because the object cannot be found)

BAIL: MMS(7924): synccoreimp.cpp(118): 0x80230405 (The operation failed because the object cannot be found)

BAIL: MMS(7924): synccoreimp.cpp(5842): 0x80230405 (The operation failed because the object cannot be found)

BAIL: MMS(7924): synccoreimp.cpp(2224): 0x80230405 (The operation failed because the object cannot be found)

ERR: MMS(7924): synccoreimp.cpp(2240): 0x80230405 - CS to MV to CS synchronization failed 0x80230405: [U12345678]

BAIL: MMS(7924): synccoreimp.cpp(2076): 0x80230405 (The operation failed because the object cannot be found)

ERR: MMS(7924): syncmonitor.cpp(2502): SE: Rollback SQL transaction for: 0x80230405

MMS(7924): SE: CS image begin

MMS(7924): <cs-object cs-dn="U12345678" id="{3FDD7506-5A93-4572-A60C-D22F4F46BEFC}" object-type="person">

  <unapplied-export>

    <delta operation="none" dn="U12345678">

      <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>

    </delta>

  </unapplied-export>

  <escrowed-export>

    <delta operation="none" dn="U12345678">

      <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>

    </delta>

  </escrowed-export>

  <unconfirmed-export>

    <delta operation="none" dn="U12345678">

      <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>

    </delta>

  </unconfirmed-export>

  <pending-import>

    <delta operation="none" dn="U12345678">

      <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>

    </delta>

  </pending-import>

  <synchronized-hologram>

    <entry dn="U12345678">

      <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>

      <primary-objectclass>person</primary-objectclass>

      <objectclass>

        <oc-value>person</oc-value>

      </objectclass>

      <attr name="DepartmentId" type="string" multivalued="false">

        <value>4321</value>

      </attr>

      <attr name="EmployeeId" type="string" multivalued="false">

        <value>U12345678</value>

      </attr>

      <attr name="UserId" type="string" multivalued="false">

        <value>U12345678</value>

      </attr>

    </entry>

  </synchronized-hologram>

  <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>

  <connector>1</connector>

  <connector-state>normal</connector-state>

  <seen-by-import>1</seen-by-import>

  <rebuild-in-progress>0</rebuild-in-progress>

  <obsoletion>0</obsoletion>

  <need-full-sync>0</need-full-sync>

  <placeholder-parent>0</placeholder-parent>

  <placeholder-link>0</placeholder-link>

  <placeholder-delete>0</placeholder-delete>

  <pending>0</pending>

  <ref-retry>0</ref-retry>

  <rename-retry>0</rename-retry>

  <sequencers>

    <current>

      <batch-number>31008</batch-number>

      <sequence-number>1529595</sequence-number>

    </current>

    <unapplied>

      <batch-number>31008</batch-number>

      <sequence-number>1529595</sequence-number>

    </unapplied>

    <original>

      <batch-number>31008</batch-number>

      <sequence-number>1529595</sequence-number>

    </original>

  </sequencers>

  <import-delta-operation>none</import-delta-operation>

  <export-delta-operation>none</export-delta-operation>

  <pending-ref-delete>0</pending-ref-delete>

  <ma-id>{410910BB-AACB-41A5-B622-7F40F246D26C}</ma-id>

  <ma-name>SQL - Roles</ma-name>

  <partition-id>{BA290343-1315-40EF-96AF-32A2D0069B36}</partition-id>

  <import-errordetail first-occurred="2012-10-17 06:45:38.283" date-occurred="2012-10-18 07:08:11.347" retry-count="16" error-type="unexpected-error">

    <import-status>

    </import-status>

  </import-errordetail>

  <mv-link lineage-id="{F434C74C-B9B8-4BCD-858B-B221447E1FC7}" lineage-type="provisioning-rules" lineage-time="2012-09-08 07:14:07.080">{9FC0E55D-F64C-4237-9255-8B3C9756C4CE}</mv-link>

  <last-import-delta-time>2012-10-18 06:14:51.587</last-import-delta-time>

  <last-export-delta-time>2012-10-16 07:36:12.437</last-export-delta-time>

</cs-object>

MMS(7924): SE: CS image end

Microsoft Identity Integration Server 3.3.0118.0"

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.