Tuesday, October 7, 2014

Network impact assessment (NIA): FIM 2010 AD MA LDAP traffic throughput

I just performed a Wireshark capture between the FIM and several Active Directory domains to try to validate whether the numbers in the MIIS 2003 Capacity Planning whitepaper are (a) still relevant to FIM 2010 and (b) actually relevant to AD connected data sources.  The whitepaper mentions that the test results show throughput between MIIS and its SQL database.  So, it doesn't specifically say that the numbers are relevant to connected data sources, but it does mention them in the recommended best practices section; and this is confusing.

So, I'm here to clear up the confusion.  And I can summarize by saying, yes, the numbers in the whitepaper are relevant to AD connected data sources.  Here are my findings:

Average LDAP traffic throughput 1.2 Mbps
Peak LDAP traffic throughput 6 Mbps

These findings seem to verify the claim from the MIIS 2003 capacity planning whitepaper that a 5 Mbps bandwidth is the bare minimum (for connected data sources). The sections below show more details about the tests.

CORP_ADMA Full import
object_type count
computer 82
contact 2072
container 208
domainDNS 1
foreignSecurityPrincipal 21
group 1787
NULL 82
organizationalUnit 616
user 10020
TOTAL objects 14889

1.14 Mbps

28287944 total bytes

28287944 / 1024^2 = 26.9775 MB

28287944 * 8 = 226303552 bits

226303552 / 198.559 = 1.1397E6 bps
6e6 bits/tick @ 1 sec/tick

6e6/1e6=6 Mbps peak




EXCH_ADMA Full import
object_type count
contact 3365
container 107
domainDNS 1
group 280
NULL 57
organizationalUnit 163
user 6740
TOTAL objects 10713

1.264 Mbps

19285496 total bytes

19285496 / 1024^2 = 18.3921 MB

5.5e6 bits/tick @ 1 sec/tick
5.5e6/1e6=5.5 Mbps peak



WEB_ADMA Full import
object_type count
container 82
domainDNS 1
group 45
NULL 8
organizationalUnit 3
user 2184
TOTAL objects 2323

1.197 Mbps

2611566 total bytes

2611566 / 1024^2 = 2.4906 MB

5e6 bits/tick @ 1 sec/tick
5e6/1e6=5 Mbps peak


Thursday, August 21, 2014

How to: Create anchor for delta import record in ECMA 2.0 GetImportEntries

#region FAIL http://social.technet.microsoft.com/Forums/en-US/a2bc4a62-a987-444a-9382-7a37ba49fcf8/getting-missingchangetype-error-during-delta-import-on-callbased-ecma2?forum=ilm2
//csentry.DN = row["dn"].ToString().Trim();
#endregion
/// Lesson learned: DO NOT name one of the columns 'DN'.
/// http://c--shark.blogspot.com/2014/08/lesson-learned-ecma2-do-not-name-one-of.html
csentry.AnchorAttributes.Add(
    AnchorAttribute.Create("SQL_DN", row["dn"].ToString().Trim()));

Lesson learned: ECMA2: Do NOT name one of the schema columns 'DN'

Here are my notes on troubleshooting errors with an ECMA 2.0. Turns out, the errors were due to the fact that one of the columns was named 'DN'. As you can imagine, troubleshooting this was maddening...
·        ECMA2 connector to SQL Server
·        One of the column names was 'DN'
·        In GetSchema:
if (string.Equals(key, "dn"StringComparison.CurrentCultureIgnoreCase))
{
    personType.Attributes.Add(SchemaAttribute.CreateAnchorAttribute(key, AttributeType.String));
}
·        Upon full import (see event log below):
o   missing-anchor-component
o   invalid-attribute-value
o   staging-error
·        GetImportEntries taken from MSFT example:
public GetImportEntriesResults GetImportEntries(GetImportEntriesRunStep importRunStep)
{
    //
    // In our sample below, we create objects using a fixed format.
    //
    GetImportEntriesResults importReturnInfo;
    List<CSEntryChange> csentries = new List<CSEntryChange>();
    // Full Import.
    if (OperationType.Full == m_ImportType)
    {
        // Object 1
        CSEntryChange csentry1 = CSEntryChange.Create();
        csentry1.ObjectModificationType = ObjectModificationType.Add;
        csentry1.ObjectType = "person";
        //csentry1.DN = "CN=Object1,CN=Users,DC=contoso,DC=com";
        csentry1.AttributeChanges.Add(AttributeChange.CreateAttributeAdd("DN""1"));
        //csentry1.AttributeChanges.Add(AttributeChange.CreateAttributeAdd("USERNAME", "Object1"));
        //csentry1.AttributeChanges.Add(AttributeChange.CreateAttributeAdd("email", "Object1@contoso.com"));
        //csentry1.AttributeChanges.Add(AttributeChange.CreateAttributeAdd("multiAttr", new List<object>() { "value1", "value1", "value3" }));
        csentries.Add(csentry1);
        // Object 2
        CSEntryChange csentry2 = CSEntryChange.Create();
        csentry2.ObjectModificationType = ObjectModificationType.Add;
        csentry2.ObjectType = "person";
        //csentry2.DN = "CN=Object2,CN=Users,DC=contoso,DC=com";
        csentry2.AttributeChanges.Add(AttributeChange.CreateAttributeAdd("DN""2"));
        //csentry2.AttributeChanges.Add(AttributeChange.CreateAttributeAdd("USERNAME", "Object2"));
        //csentry2.AttributeChanges.Add(AttributeChange.CreateAttributeAdd("email", "Object2@contoso.com"));
        //csentry2.AttributeChanges.Add(AttributeChange.CreateAttributeAdd("multiAttr", new List<object>() { "value4", "value5", "value6" }));
        csentries.Add(csentry2);
    }
    // Create the result
    importReturnInfo = new GetImportEntriesResults();
    importReturnInfo.MoreToImport = false;
    importReturnInfo.CSEntries = csentries;
    // Also, GetImportEntriesResults.CustomData can be used for retaining necessary data
    // for performing a subsequent import.
    // A typical case of this might be the watermark used
    // for delta import.
    //
    return importReturnInfo;
}
·        Event log:
FIMSynchronizationService
The server encountered an unexpected error while performing an operation for a management agent.
 "Microsoft.MetadirectoryServices.Impl.InternalError: 0x80230301
   at Microsoft.MetadirectoryServices.Impl.ScriptHost.ThrowExceptionFromHRESULT(Int32 hr)
   at Microsoft.MetadirectoryServices.Impl.Ecma2ConversionServices.AddAttributeToDImage(CDImage* pdimage, String attributeName, AttributeModificationType attributeModificationType, IList`1 attributeValueChanges, Int32 escapeReferenceDNValues)
   at Microsoft.MetadirectoryServices.Impl.Ecma2ConversionServices.ConvertToDImage(CSEntryChange csEntryChange, CDImage** ppDImage, Int32 escapeReferenceDNValues)
   at Microsoft.MetadirectoryServices.Impl.ScriptHost.InvokeExtMA_ImportEntry(UInt32 cBatchSize, UInt16* pcszCustomData, UInt32 cFullObject, _OCTET* rgoctFullObject, UInt32* rgomodt, UInt32* pcpcszChangedAttributes, UInt16*** prgpcszChangedAttributes, Int32 fIsDNStyleNone, UInt16** ppszUpdatedCustomData, _OCTET* rgoctCSImage, Int32* rgextec, UInt16** rgpszErrorName, UInt16** rgpszErrorDetail, Int32* pfMoreToImport)"
FIMSynchronizationService
The server encountered an unexpected error in the synchronization engine:
 "BAIL: MMS(8916): d:\bt\32669\private\source\miis\server\sync\syncstage.cpp(595): 0x80230302 (The image or delta doesn't have an anchor.)
Forefront Identity Manager 4.1.3559.0"
FIMSynchronizationService
The server encountered an unexpected error in the synchronization engine:
 "BAIL: MMS(6992): d:\bt\32669\private\source\miis\ma\xml\ma.cpp(5012): 0x80230005 (Management agent encountered an error while processing an object.)
BAIL: MMS(6992): d:\bt\32669\private\source\miis\server\sync\syncstage.cpp(595): 0x80230302 (The image or delta doesn't have an anchor.)
Forefront Identity Manager 4.1.3559.0"
FIMSynchronizationService
The server encountered an unexpected error in the synchronization engine:
 "BAIL: MMS(13268): d:\bt\32669\private\source\miis\ma\xml\ma.cpp(5078): 0x80230005 (Management agent encountered an error while processing an object.)
BAIL: MMS(13268): d:\bt\32669\private\source\miis\shared\ldaputils\dnutils.cpp(580): 0x80070057 (The parameter is incorrect.): Error parsing DN 99999: DN terminates in attribute name
BAIL: MMS(13268): d:\bt\32669\private\source\miis\ma\extensible\extensible.cpp(2057): 0x80070057 (The parameter is incorrect.)
BAIL: MMS(13268): d:\bt\32669\private\source\miis\server\sync\syncstage.cpp(595): 0x80230302 (The image or delta doesn't have an anchor.)
Forefront Identity Manager 4.1.3559.0"
FIMSynchronizationService
The server encountered an unexpected error in the synchronization engine:
 "BAIL: MMS(8916): d:\bt\32669\private\source\miis\ma\xml\ma.h(734): 0x80230005 (Management agent encountered an error while processing an object.)
BAIL: MMS(8916): d:\bt\32669\private\source\miis\ma\xml\ma.cpp(4358): 0x80230005 (Management agent encountered an error while processing an object.)
BAIL: MMS(8916): d:\bt\32669\private\source\miis\ma\xml\ma.cpp(5240): 0x80230005 (Management agent encountered an error while processing an object.)
BAIL: MMS(8916): d:\bt\32669\private\source\miis\server\sync\syncstage.cpp(595): 0x80230302 (The image or delta doesn't have an anchor.)
Forefront Identity Manager 4.1.3559.0"

Monday, July 22, 2013

FIM metaverse SQL query: expected rules list

 

set transaction isolation level read uncommitted

 

select

     mvpers.displayName as [User Display Name]

     ,mvpers.accountName as [User Account Name]

    ,mvere.displayName as [ERE Display Name]

    ,mvere.object_type

    ,link.*

from dbo.mms_mv_link link

join dbo.mms_metaverse mvpers

on link.object_id = mvpers.object_id

full outer join dbo.mms_metaverse mvere

on reference_id = mvere.object_id

where attribute_name = 'expectedRulesList'

 

Sunday, July 14, 2013

Outbound System Scoping Filter: "Contains" doesn't work on multi-valued attributes (and why no "Is Present"?)

I was excited to finally have a scenario in which I could use the new Outbound System Scoping Filter, but sadly its limitations put the kibosh on that idea.  I was surprised to discover (empirically, of course) that you cannot use a "contains" operator on a multi-valued attribute.  What's more, there is no "is present" operator at all.  Anyone else frustrated by FIM's lack of consistency? </rant>

Anyway, here are the tests that I ran, in case you're interested:


Test
Scoping filter
Result
Single-valued attribute equals
employeeID EQUAL 100000
OK
Multi-valued attribute contains string
Roles CONTAINS MyRoleName
FAIL
(Not Applied)
Attempt at multi-valued attribute contains string
Roles STARTSWITH MyRoleName
FAIL
(Not Applied)
Another attempt at multi-valued attribute contains string
Roles EQUAL MyRoleName
FAIL
(Not Applied)
Desperate attempt at multi-valued attribute contains string
Roles STARTSWITH %MyRoleName%
FAIL
(Not Applied)
Attempt at "Is Present"
employeeID NOTEQUAL NO SUCH EMPID
FAIL
(Applied when employeeID is not present)
Another attempt at "Is Present"
employeeID STARTSWITH %
FAIL
(Not Applied)
Desperate attempt at "Is Present"
employeeID GREATERTHAN 0
SUCCESS
100000: Applied
A100000: Applied
<empty>: Not Applied

And here's the explanation from Andreas on the forum:

"That is not supposed to work. The scoping filter is only intended for string values. It isn't documented because I didn't think of documenting it."

Tuesday, July 9, 2013

FIM metaverse SQL query - employeeID contributing MA


set transaction isolation level read uncommitted

SELECT TOP 1000
    --lg.[object_id]
    mv.object_type
      --,lg.[displayName]
      --,lg.[employeeID]
      --,lg.[HR_USERID]
      --,lg.[HR_Joined]

    ,mv.employeeID as [employeeID]
    ,ma_empid.ma_name as [employeeID MA]

    ,mv.displayName as displayName
    ,ma_disp.ma_name as [displayName MA]

    ,mv.HR_USERID as HR_USERID
    ,ma_uid.ma_name as [HR_USERID MA]

    ,mv.HR_Joined as HR_Joined
    ,ma_hrj.ma_name as [HR_Joined MA]

  FROM [FIMSynchronizationService].[dbo].[mms_metaverse_lineageguid] lg
  join [FIMSynchronizationService].dbo.mms_metaverse mv
  on lg.object_id = mv.object_id

  left join [FIMSynchronizationService].[dbo].[mms_lineage_cross_reference] cr_empid
  on cr_empid.lineage_id = lg.employeeID
  left join [FIMSynchronizationService].[dbo].[mms_management_agent] ma_empid
  on ma_empid.ma_id = cr_empid.ma_id

  left join [FIMSynchronizationService].[dbo].[mms_lineage_cross_reference] cr_disp
  on cr_disp.lineage_id = lg.displayName
  left join [FIMSynchronizationService].[dbo].[mms_management_agent] ma_disp
  on ma_disp.ma_id = cr_disp.ma_id

  left join [FIMSynchronizationService].[dbo].[mms_lineage_cross_reference] cr_uid
  on cr_uid.lineage_id = lg.HR_USERID
  left join [FIMSynchronizationService].[dbo].[mms_management_agent] ma_uid
  on ma_uid.ma_id = cr_uid.ma_id

  left join [FIMSynchronizationService].[dbo].[mms_lineage_cross_reference] cr_hrj
  on cr_hrj.lineage_id = lg.HR_Joined
  left join [FIMSynchronizationService].[dbo].[mms_management_agent] ma_hrj
  on ma_hrj.ma_id = cr_hrj.ma_id

  where object_type = 'person'
  and (
        ma_empid.ma_name = 'FIM MA'
        or mv.employeeID is null
        --or mv.HR_Joined is null
    )


Wednesday, June 26, 2013

Dude, where's my flow?

Got burned by FIM's wacky function behavior again, so I thought I'd actually create some documentation on it.

I set up a pretty common flow: Name = First Middle Last

However, when the middle name is empty, I don't want two blanks between first and last name.  So, how's this for a custom expression?  (Note that I'm assuming that first and last name are always present.)

Trim(givenName+" "+HR_MIDDLE_NAME)+" "+sn

Submit, import, sync.  Now I run a full sync from my target MA.  No errors.  We're good, right?  Not so fast; let's check the attribute flow.  No flow into Name.

Okay, so let's run a full sync preview on one of the connectors.  The status of the flow is Error.  (Good luck finding an error message.)
  • Full sync preview
    • Connector updates
      • Export attribute flow
        • Outbound sync rules
Status
MV attribute
Mapping type
Data source attribute
Initial value
Final value
Error

sync-rule-mapping - expression
NAME

(Unchanged)

So what happened?  Evidently, FIM doesn't like it when you concatenate an attribute that isn't present on the metaverse object.  (Has a funny way of showing it though, doesn't it?)

For my reference and yours, here are the results of other experiments that I ran on this flow.
  • HR_MIDDLE_NAME is not present in MV
  • (Assume givenName and sn are always present)

Description
Function
Flow definition
Result/status
MV attribute not present, embedded expr in Trim
CustomExpression
Trim(givenName+" "+HR_MIDDLE_NAME)+" "+sn
Error
Same expression but using concat feature in portal interface
Concat
Trim(CustomExpression(givenName+" "+HR_MIDDLE_NAME))+" "+sn
Error
Added IsPresent check and removed Trim
CustomExpression
givenName+IIF(IsPresent(HR_MIDDLE_NAME)," "+HR_MIDDLE_NAME,"")+" "+sn
Success
Trim function with no assumptions
CustomExpression
Trim(
""+
IIF(IsPresent(givenName),givenName,"")+
" "+
IIF(IsPresent(HR_MIDDLE_NAME),HR_MIDDLE_NAME,"")+
" "+
IIF(IsPresent(sn),sn,"")
)
Parse failure:
Parameter of function Trim does not match
(Test embedded expr in Trim)
CustomExpression
Trim(givenName+" "+sn)
(Success)
(Test MV attribute not present in Trim)
CustomExpression
givenName+" "+HR_MIDDLE_NAME+" "+sn
(Error)