Fixed! FIM Sync hang/crash and performance counters

Ran into an issue where the FIM Sync service was hung up pretty badly.  Eventually, I was able to fix it, and fixed the performance counters along the way (although I'm not entirely sure the two had a direct correlation).

• At 8AM, Full sync had been running since 3:49AM
• SQL Server MA
• miiserver.exe was unresponsive
• Tried to stop run profile --> nothing
• Tried to stop service --> nothing
• Had to kill the process
• Afterwards, service randomly crashed
• Ran Storechk.exe --> no issues
• Manually checked for broken CS-MV links --> no issues
• Resolution
• Deleted run history
• Rebuilt indexes on FIMSynchronizationService
• Reinstalled Performance counters

Reinstall performance counters

The server encountered an unexpected error creating performance counters for management agent "Foobar". Performance counters will not be available for this management agent.

1. Tried to reinstall performance counters --> no luck
   
   C:\WINDOWS\system32>lodctr.exe "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Bin\mmsperf.ini"
   
   
   Counters for FIMSynchronizationService are already installed
   
   
2. Queried FIM Sync performance counters, and noticed that they were disabled.
   
   C:\WINDOWS\system32>lodctr.exe /q:FIMSynchronizationService
   Performance Counter ID Queries [PERFLIB]:
   Base Index: 0x00000737 (1847)
   Last Counter Text ID: 0x0000265A (9818)
   Last Help Text ID: 0x0000265B (9819)
   
   
   [FIMSynchronizationService] Performance Counters (Disabled)
   DLL Name: C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Bin\mmsperf.dll
   Open Procedure: PrfData_Open
   Collect Procedure: PrfData_Collect
   Close Procedure: PrfData_Close
   First Counter ID: 0x000024AE (9390)
   Last Counter ID: 0x0000251C (9500)
   First Help ID: 0x000024AF (9391)
   Last Help ID: 0x0000251D (9501)

   
   
3. Tried to enable FIM Sync perf counters --> no change
   
   
   C:\WINDOWS\system32>lodctr.exe /e:FIMSynchronizationService
   
   
   C:\WINDOWS\system32>lodctr.exe /q:FIMSynchronizationService
   Performance Counter ID Queries [PERFLIB]:
   Base Index: 0x00000737 (1847)
   Last Counter Text ID: 0x0000265A (9818)
   Last Help Text ID: 0x0000265B (9819)
   
   
   [FIMSynchronizationService] Performance Counters (Enabled)
   DLL Name: C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Bin\mmsperf.dll
   Open Procedure: PrfData_Open
   Collect Procedure: PrfData_Collect
   Close Procedure: PrfData_Close
   First Counter ID: 0x000024AE (9390)
   Last Counter ID: 0x0000251C (9500)
   First Help ID: 0x000024AF (9391)
   Last Help ID: 0x0000251D (9501)
   
   
4. Tried to unload FIM Sync perf counters --> couldn't get it to work
   
   C:\WINDOWS\system32>unlodctr \\SERVER01 FIMSynchronizationService
   
   Usage:
   
   UNLODCTR [\\computername] driver
   
   
   
5. Deleted registry key
   
   
   [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\FIMSynchronizationService\Performance]
   
   
   
6. Manually created 'Performance' registry key.
   
   
7. Reinstalled performance counters --> FIXED!
   
   
   lodctr.exe "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Bin\mmsperf.ini"

References

http://stackoverflow.com/questions/140149/deleting-windows-performance-counter-categories

http://www.identitytrench.com/2010/08/sync-performance-counters.html