faster way to get AD memberships for millions of AD groups in a network with multiple trusted forests and domains

Question

First, I cannot get into why I need this data and I cannot get into specifics about the network. You'll have to trust me there is no other way to get this data other than a PowerShell script to run LDAP queries.

I am working with a network that has multiple forests and multiple domains. There is a trust between all the forests. I am logged into one domain on one of the forests but because of the trust I can query all of them.

I have a CSV file with millions of AD groups. I need to find all the direct members of everyone of the millions of AD groups. A lot of memberships are cross-domain which means I cannot just use the member property of the AD group and have to, instead, query every domain and check for memberOf.

I have a PowerShell script that gets this data. For various reasons I cannot share my code but here is what it does:

1. creates an array of System.DirectoryServices.DirectorySearcher objects for all of my domains
2. iterate through the CSV file that has a list of every AD group and its DN
3. for each DN, loop over the DirectorySearcher array and find all objects that are a memberOf the AD group in that DirectorySearcher ((memberOf=$adGroupDN))

The code works. But since I'm dealing with an input list with millions of AD groups the script is awfully slow. Based on my test run calculations it will take more than 2 weeks to get all of the data I need.

I'm wondering if there is a better/faster way to do this?

I thought maybe I could use threading or something but I am not sure if that will help nor am I sure where to start.

Any advise is greatly appreciated.

Adding some additional details...

• My input list are millions of unique group DNs
• I have multiple different forests/domains
• My input group DNs are spread across all the forests/domains
• The groups that are in my input list of group DNs span different forests/domains (domain1\group1 from my input list has domain2\group2 as a member)
• I need to get a complete list of every group that is in the groups from my input list
• Because of cross-domain memberships I cannot rely on the member attribute of the my input groups. The only way I know to get it is query every DC/domain for all groups that are are memberOf the groups from my input list.
• I can only use PowerShell
• I do not have the ActiveDirectory module and can only use the .NET DirectorySearcher

• At a high level my code looks like this:

  $arrayOfDirectorySearcherObjectsForEachDCInMyNetwork = ... code to create an array of System.DirectoryServices.DirectorySearcher objects, one for each DC/domain in my network
  
  Foreach ($groupDN in $inputListOfUniqueGroupDNs)
  {
      Foreach ($domain in $arrayOfDirectorySearcherObjectsForEachDCInMyNetwork)
      {
          ...
  

• The only way I can think of making it faster is to multi-thread the second for loop where it queries multiple DCs/domains at the same time using runspaces but I cannot figure out how to do this...

Answer 1

Running the script on a domain controller would give you a slight advantage, if it's an option. But otherwise multi-threading is likely your best bet.

Look into using Start-Job. There's an example here.

That said, I question this:

A lot of memberships are cross-domain which means I cannot just use the member property of the AD group and have to, instead, query every domain and check for memberOf.

Group scope is important here. If all of your groups are Universal, then either way shouldn't make a difference (whether you look at member on the group or memberOf on the users).

But it's important to note that memberOf will not show Domain Local groups groups on a different domain (even in the same forest).

The member attribute on a group is always the authoritative source for the members. Yes, getting the details of an account on a trusted domain is a little tougher, but it can be done.

Here is a PowerShell function that will pull the "domain\username" of every member of a group, including those in nested groups.

function OutputMembers {
    param([string] $groupDn)

    foreach ($m in ([ADSI]("LDAP://" + $groupDn)).member) {
        $member = [ADSI]("LDAP://" + $m)
        $member.objectClass
        if ($member.objectClass -eq "group") {
            #this member is a group so pull the members of that group
            OutputMembers $member.distinguishedName
        } else {
            #"msDS-PrincipalName" is not loaded by default, so we have to tell it to get it
            $member.Invoke("GetInfoEx", @("msDS-PrincipalName"), 0)
            if ([string]::IsNullOrEmpty($member."msDS-PrincipalName")) {
                #member is on a trusted domain, so we have to go look it up
                $sid = New-Object System.Security.Principal.SecurityIdentifier ($member.objectSid[0], 0)
                $sid.Translate([System.Security.Principal.NTAccount]).value
            } else {
                $member."msDS-PrincipalName"
            }
        }
    }
}

With that, you call that function with the distinguishedName of each group, like:

OutputMembers "CN=MyGroup,OU=Groups,DC=domain,DC=com"

Answer 2

I presume that the performance issue in step 3 as that presumably has an embedded loop (that might even be recursive if you look for indirect group memberships as well):

Foreach ($UserDN in $UserDNs) {
    ...
    Foreach ($GroupDN in $GroupDNs) {
        ...

Everything in the inner loop is very important for the performance of your script as that will be invoked $UserDNs.Count * $GroupDNs.Count times!
I suspect that there are a lot of redundant LDAP queries in the inner loop (for user that are in the same group) and therefore focus on that and build a kind of custom caching of every redundant query to the server to overcome this. Something like:

$MemberCache = @{}

Function GetMembers([String]$GroupDN) {
    If (!$MemberCache.ContainsKey($GroupDN)) {
        $MemberCache[$GroupDN] = @{}    #HashTables are much faster then using the contains method on an array
        # retrieve all members of the AD group in that DirectorySearcher
        ForEach ($Member in $Members) {$MemberCache[$GroupDN].$Member = $True}
    }
    $MemberCache[$GroupDN]
}

Function IsMember([String]$DN, [String]$GroupDN) {
    (GetMembers($GroupDN)).ContainsKey($DN)
}

The general idea is that you should not remotely redo the "find all objects that are a memberOf the AD group in that DirectorySearcher ((memberOf=$adGroupDN))" for the same $adGroupDN (any group you already queried before) but retrieve the required information from a local hash table (cache).

Answer 3

There are several optimizations can be done here. These optimizations are not related to Powershell, but to the algorithm itself.

1. Powershell is not designed to perform such kind of tasks. C\C++ or at least C# should be used instead.
2. Create and maintain connection to RootDse object of each global catalog that you're querying information from until all job is done. In this case all AD queries will use one single cached connection to AD, which significantly increases performance.
3. Thumbs up to iRon. Create a cache for all queried groups. For example, if you queried membership for group A and group A is a member of group B, there is no need to query memberhip of group A again. Of course, in your case you just cannot simply store all membership in memory, so you need to create some kind of storage where membership will be saved
4. Read groups from CSV in parallel. Make several threads. To combine it with group cache you need to operate 2 caches. One for already queried groups and the other one for pending groups, that threads are querying at the moment (to avoid double query the same group in different threads). These caches should be thread safe of course. If one thread sees, that another is querying the group, this thread can skip current group and query any other, and return back later.
5. For foreign security principals (users and groups from trusted domains) use SID binding. You can extract sid for FSP from Member DN. Direct binding works much faster than DirectorySearcher

Be aware of:

1. Thumbs up to Gabriel Luci. You need to query member, not memberOf attribute,
2. Don't forget about nested groups (group A -> group B -> user U). User is a member of group B as well a group A. This rule applies to trusted domain groups as well
3. Groups may be members of each other. E. g. A -> B -> C -> A or even A -> B -> A. You have to handle this in your script