Thursday, February 28, 2013

Fixed! FIM Sync hang/crash and performance counters

Ran into an issue where the FIM Sync service was hung up pretty badly.  Eventually, I was able to fix it, and fixed the performance counters along the way (although I'm not entirely sure the two had a direct correlation).
  • At 8AM, Full sync had been running since 3:49AM
    • SQL Server MA
  • miiserver.exe was unresponsive
    • Tried to stop run profile --> nothing
    • Tried to stop service --> nothing
  • Had to kill the process
  • Afterwards, service randomly crashed
  • Ran Storechk.exe --> no issues
  • Manually checked for broken CS-MV links --> no issues
  • Resolution
    • Deleted run history
    • Rebuilt indexes on FIMSynchronizationService
    • Reinstalled Performance counters

Reinstall performance counters

The server encountered an unexpected error creating performance counters for management agent "Foobar". Performance counters will not be available for this management agent.
  1. Tried to reinstall performance counters --> no luck

    C:\WINDOWS\system32>lodctr.exe "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Bin\mmsperf.ini"

    Counters for FIMSynchronizationService are already installed

  2. Queried FIM Sync performance counters, and noticed that they were disabled.

    C:\WINDOWS\system32>lodctr.exe /q:FIMSynchronizationService
    Performance Counter ID Queries [PERFLIB]:
    Base Index: 0x00000737 (1847)
    Last Counter Text ID: 0x0000265A (9818)
    Last Help Text ID: 0x0000265B (9819)

    [FIMSynchronizationService] Performance Counters (Disabled)
    DLL Name: C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Bin\mmsperf.dll
    Open Procedure: PrfData_Open
    Collect Procedure: PrfData_Collect
    Close Procedure: PrfData_Close
    First Counter ID: 0x000024AE (9390)
    Last Counter ID: 0x0000251C (9500)
    First Help ID: 0x000024AF (9391)
    Last Help ID: 0x0000251D (9501)

  3. Tried to enable FIM Sync perf counters --> no change

    C:\WINDOWS\system32>lodctr.exe /e:FIMSynchronizationService

    C:\WINDOWS\system32>lodctr.exe /q:FIMSynchronizationService
    Performance Counter ID Queries [PERFLIB]:
    Base Index: 0x00000737 (1847)
    Last Counter Text ID: 0x0000265A (9818)
    Last Help Text ID: 0x0000265B (9819)

    [FIMSynchronizationService] Performance Counters (Enabled)
    DLL Name: C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Bin\mmsperf.dll
    Open Procedure: PrfData_Open
    Collect Procedure: PrfData_Collect
    Close Procedure: PrfData_Close
    First Counter ID: 0x000024AE (9390)
    Last Counter ID: 0x0000251C (9500)
    First Help ID: 0x000024AF (9391)
    Last Help ID: 0x0000251D (9501)

  4. Tried to unload FIM Sync perf counters --> couldn't get it to work

    C:\WINDOWS\system32>unlodctr \\SERVER01 FIMSynchronizationService

    Usage:

    UNLODCTR [\\computername] driver

  5. Deleted registry key

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\FIMSynchronizationService\Performance]

  6. Manually created 'Performance' registry key.

  7. Reinstalled performance counters --> FIXED!

    lodctr.exe "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Bin\mmsperf.ini"

References