Monday, November 9, 2015

How to: fix unexpected-error (not supported)

Obligatory disclaimer: This method of fixing your FIM Sync database manually is not supported; use at your own risk! As always, back up your FIM Sync database before attempting any such operation.

Now that we've got that out of the way, I wouldn't be blogging about this if it hadn't worked.

A little bit of background: I troubleshot this error back in 2012, so I'm working from my notes. I believe the 'unexpected-error' showed up in the run history, but I can't remember if FIM Sync actually crashed; maybe this will help someone out there anyway. At the time, I recall troubleshooting an actual crash of miiserver.exe, so unfortunately I don't remember if we had tried storechk.exe before I fixed the database with this hack.  Anyway, here goes...

I've pasted the event log entry at the bottom of this post for reference.  General steps to fix are:

1. Enumerate orphaned CS-MV links
2. Save off orphaned links
3. Delete CS records
4. Delete links
5. Full import from affected MAs
6. Full syncs as necessary


1. Enumerate orphaned CS-MV links

select cs.ma_id, ma.[ma_name] ,count(*) as [count],min([initial_import_error_date]) as [min initial import error date]
from dbo.mms_connectorspace cs
join [dbo].[mms_management_agent] ma
on ma.[ma_id] = cs.[ma_id]
join (

SELECT [mv_object_id]
,mv.[object_id] as [mv.object_id]
      ,[cs_object_id]
      ,[lineage_id]
      ,[lineage_date]
  FROM [FIMSynchronizationService].[dbo].[mms_csmv_link] csmv

full outer join [FIMSynchronizationService].[dbo].[mms_metaverse] mv

on mv.[object_id] = csmv.[mv_object_id]
where mv.[object_id] is null
)b
on b.cs_object_id = cs.object_id
group by cs.ma_id, ma.[ma_name]
order by count(*) desc

ma_name
count
min initial import error date
SQL - Roles
764
10/17/12 6:43 AM
SQL - Virtual
348
NULL
SQL - 3rd party
277
10/16/12 8:16 PM
SQL - Principal
277
10/16/12 8:09 PM
AD-LDS - Partners
251
10/16/12 9:06 PM
SQL - Groups
238
10/16/12 8:07 PM
SQL - SSO
238
10/16/12 8:08 PM
SQL - Mapping
2
10/17/12 5:27 AM
SQL - User gen
2
10/17/12 5:43 AM
AD - Primary
2
10/17/12 3:51 AM
ECMA - Java
1
10/17/12 3:40 AM
ECMA - Web services
1
10/17/12 3:47 AM

2. Save off orphaned links
SELECT [mv_object_id]
,mv.[object_id] as [mv.object_id]
      ,[cs_object_id]
      ,[lineage_id]
      ,[lineage_date]
into #wehackedit
  FROM [FIMSynchronizationService].[dbo].[mms_csmv_link] csmv
full outer join [FIMSynchronizationService].[dbo].[mms_metaverse] mv
on mv.[object_id] = csmv.[mv_object_id]
where mv.[object_id] is null

3. Delete CS records

delete from dbo.mms_connectorspace
where object_id in
(
select cs_object_id from #wehackedit
)

4. Delete links

delete from dbo.mms_csmv_link
where mv_object_id in
(
select [mv_object_id] from #wehackedit
)

Event log

Event Type:    Error
Event Source:    MIIServer
Event Category:    Server
Event ID:    6301
Date:        10/18/2012
Time:        12:17:53 AM
User:        N/A
Computer:    FIMSYNC01
Description:
The server encountered an unexpected error in the synchronization engine:

"BAIL: MMS(7924): sproc.cpp(1198): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): imgbldr.cpp(1216): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): imgbldr.cpp(1078): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): imgbldr.cpp(3145): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): mvobj.cpp(199): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): nsmvimp.cpp(274): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): csobj.cpp(2062): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): synccore.cpp(555): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): synccoreimp.cpp(118): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): synccoreimp.cpp(5842): 0x80230405 (The operation failed because the object cannot be found)
BAIL: MMS(7924): synccoreimp.cpp(2224): 0x80230405 (The operation failed because the object cannot be found)
ERR: MMS(7924): synccoreimp.cpp(2240): 0x80230405 - CS to MV to CS synchronization failed 0x80230405: [U12345678]
BAIL: MMS(7924): synccoreimp.cpp(2076): 0x80230405 (The operation failed because the object cannot be found)
ERR: MMS(7924): syncmonitor.cpp(2502): SE: Rollback SQL transaction for: 0x80230405
MMS(7924): SE: CS image begin
MMS(7924): <cs-object cs-dn="U12345678" id="{3FDD7506-5A93-4572-A60C-D22F4F46BEFC}" object-type="person">
  <unapplied-export>
    <delta operation="none" dn="U12345678">
      <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
    </delta>
  </unapplied-export>
  <escrowed-export>
    <delta operation="none" dn="U12345678">
      <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
    </delta>
  </escrowed-export>
  <unconfirmed-export>
    <delta operation="none" dn="U12345678">
      <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
    </delta>
  </unconfirmed-export>
  <pending-import>
    <delta operation="none" dn="U12345678">
      <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
    </delta>
  </pending-import>
  <synchronized-hologram>
    <entry dn="U12345678">
      <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
      <primary-objectclass>person</primary-objectclass>
      <objectclass>
        <oc-value>person</oc-value>
      </objectclass>
      <attr name="DepartmentId" type="string" multivalued="false">
        <value>4321</value>
      </attr>
      <attr name="EmployeeId" type="string" multivalued="false">
        <value>U12345678</value>
      </attr>
      <attr name="UserId" type="string" multivalued="false">
        <value>U12345678</value>
      </attr>
    </entry>
  </synchronized-hologram>
  <anchor encoding="base64">EgAAAFUANgAyADAANAAyADEAMQA0AA==</anchor>
  <connector>1</connector>
  <connector-state>normal</connector-state>
  <seen-by-import>1</seen-by-import>
  <rebuild-in-progress>0</rebuild-in-progress>
  <obsoletion>0</obsoletion>
  <need-full-sync>0</need-full-sync>
  <placeholder-parent>0</placeholder-parent>
  <placeholder-link>0</placeholder-link>
  <placeholder-delete>0</placeholder-delete>
  <pending>0</pending>
  <ref-retry>0</ref-retry>
  <rename-retry>0</rename-retry>
  <sequencers>
    <current>
      <batch-number>31008</batch-number>
      <sequence-number>1529595</sequence-number>
    </current>
    <unapplied>
      <batch-number>31008</batch-number>
      <sequence-number>1529595</sequence-number>
    </unapplied>
    <original>
      <batch-number>31008</batch-number>
      <sequence-number>1529595</sequence-number>
    </original>
  </sequencers>
  <import-delta-operation>none</import-delta-operation>
  <export-delta-operation>none</export-delta-operation>
  <pending-ref-delete>0</pending-ref-delete>
  <ma-id>{410910BB-AACB-41A5-B622-7F40F246D26C}</ma-id>
  <ma-name>SQL - Roles</ma-name>
  <partition-id>{BA290343-1315-40EF-96AF-32A2D0069B36}</partition-id>
  <import-errordetail first-occurred="2012-10-17 06:45:38.283" date-occurred="2012-10-18 07:08:11.347" retry-count="16" error-type="unexpected-error">
    <import-status>
    </import-status>

  </import-errordetail>
  <mv-link lineage-id="{F434C74C-B9B8-4BCD-858B-B221447E1FC7}" lineage-type="provisioning-rules" lineage-time="2012-09-08 07:14:07.080">{9FC0E55D-F64C-4237-9255-8B3C9756C4CE}</mv-link>
  <last-import-delta-time>2012-10-18 06:14:51.587</last-import-delta-time>
  <last-export-delta-time>2012-10-16 07:36:12.437</last-export-delta-time>
</cs-object>

MMS(7924): SE: CS image end
Microsoft Identity Integration Server 3.3.0118.0"

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.