How to add foreach-object -parallel output to a dictionary/arraylist

Question

I'm trying to get all the stale branches from Azure DevOps to nag the developers to remove them. If I use this script below, I get all the results I want, but it takes ages to process.

$resultlist = New-Object System.Collections.ArrayList

function Get-StaleBranches {
    $dateString = (Get-Date).AddDays(-90).ToString("MM/dd/yyyy HH:mm:ss")
    $date = [datetime]::parseexact($dateString, 'MM/dd/yyyy HH:mm:ss', $null)
    $reposArray | ForEach-Object {
        $repo = $PSItem
        $refsUri = "url"
        $refs = (Invoke-RestMethod -Uri $refsUri -Method get -Headers $AzureDevOpsAuthenicationHeader).value
        foreach($branch in $refs){
            $splitName = $branch.name.Substring(11)
            $commitUri = $using:OrgUri + "url"
            $commits = (Invoke-RestMethod -Uri $commitUri -Method get -Headers $AzureDevOpsAuthenicationHeader).value
            $commitDate = [datetime]::parseexact($commits.author.date, 'MM/dd/yyyy HH:mm:ss', $null)
            if($commitDate -lt $date -and $splitName -notlike "develop" -and $splitName -notlike "release" -and $splitName -notlike "master")
            {
                $result = @{}
                $result.repo = $repo
                $result.branch = $splitName
                $result.date = $commitDate
                $result.author = $commits.author.name
                $result.email = $commits.author.email
                $resultlist.Add((New-Object PsObject -Property $result)) | Out-Null
            }
        }
    }
}

Get-StaleBranches

To speed it up, I tried using the foreach-object -parallel functionality like this:

$threadSafeDictionary = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()

function Get-StaleBranches {
    $dateString = (Get-Date).AddDays(-90).ToString("MM/dd/yyyy HH:mm:ss")
    $date = [datetime]::parseexact($dateString, 'MM/dd/yyyy HH:mm:ss', $null)
    $reposArray | ForEach-Object -Parallel {
        $dict = $using:threadSafeDictionary
        $repo = $PSItem
        $dict.$repo = @()
        $refsUri = "url"
        $refs = (Invoke-RestMethod -Uri $refsUri -Method get -Headers $using:AzureDevOpsAuthenicationHeader).value
        foreach($branch in $refs){
            $splitName = $branch.name.Substring(11)
            $commitUri = "url"
            $commits = (Invoke-RestMethod -Uri $commitUri -Method get -Headers $using:AzureDevOpsAuthenicationHeader).value
            $commitDate = [datetime]::parseexact($commits.author.date, 'MM/dd/yyyy HH:mm:ss', $null)
            if($commitDate -lt $using:date -and $splitName -notlike "develop" -and $splitName -notlike "release" -and $splitName -notlike "master")
            {
                $dict.$repo += [PSCustomObject]@{
                    Branch = $splitName
                    Date = $commitDate
                    Author = $commits.author.name
                    Email = $commits.author.email
                }
            }
        }
    }
}

Get-StaleBranches

However, now all branches in the dictionary are doubled. Did I make a mistake somewhere? Is there any other way to approach this?

Any help will be appreciated.

Can you run a test for me? I found the += ran slow. So I changed From : $dict.$repo = @() and $dict.$repo += [PSCustomObject]@..... TO : $dict.$repo = [System.Collections.ArrayList]::new() and $dict.$repo.Add( [PSCustomObject]@.......) — jdweng, Feb 06 '23 at 16:16
Hi, first of all, I'm now getting some numbers in console output, it goes extremely fast for first 30 seconds and then starts outputting numbers at a speed of around 4 per second until it gets to 617. Do you know what that is? But the doubled entries are no longer there, so that must've been related to the += somehow. Thank you so much. — Thyica, Feb 06 '23 at 16:42
The number is the result of the return value from the `.Add()` call getting implicitly output - see [this answer](https://stackoverflow.com/a/70928244/45375). While you can fix that with `$null = ....`, there's no good reason to use an array list to begin with - just let PowerShell collect the outputs _automatically_ for you - see [this answer](https://stackoverflow.com/a/60708579/45375). — mklement0, Feb 06 '23 at 16:45
You don't even need a `ConcurrentDictionary` if you just rely on PowerShell's implicit output behaviour. So just write that `[PSCustomObject]@{…}` literal without assigning it to anything. Collect output from the function instead: `$staleBranches = Get-StaleBranches`. — zett42, Feb 06 '23 at 16:52
Thanks @mklement0, so if I understand the 2nd link correctly, instead of the `$dict.$repo += ...` part, I should just leave the object declaration and add the object on the same level as I'm doing the `$dict.$repo = [System.Collections.ArrayList]::new()`? — Thyica, Feb 06 '23 at 16:53

mklement0 · Answer 1 · 2023-02-06T17:03:28.413

I have no explanation for your symptom, but I recommend simplifying your code as follows, which will also speed it up:

Avoid the explicit use of (synchronized) data structures and instead let PowerShell collect loop output in an array for you, as described in this answer, which works with any command output (and therefore also with ForEach-Object [-Parallel] and a foreach loop statement).

The following simplified example shows how to do that:

function Get-StaleBranches {
  # Implicitly output what the ForEach-Object -Parallel script block outputs.
  (1..5).ForEach({ 'repo' + $_ }) | ForEach-Object -Parallel {
    # Create and implicitly output a custom object for each repo.
    [pscustomobject] @{
      Repo   = $PSItem
      Results = foreach ($branch in (1..3).ForEach({ 'branch' + $_ })) { 
        # Create and implicitly output a custom object; all such objects
        # created in this `foreach` loop are collected in the .Results property.
        [pscustomobject] @{
          Branch = $branch
          Date   = Get-Date
        }
      }
    }   
  }
}

Note:

Calling Get-StaleBranches outputs a stream of (nested) [pscustomobject] instances, which, when captured in a variable, become an [object[]] array.

Display output:

Repo  Result
----  ------
repo1 {@{Branch=branch1; Date=2/6/2023 11:53:33 AM}, @{Branch=branch2; Date=2/6/2023 11:53:33 AM}, @{Branch=branch3; Date=2/6/2023 11:53:33 AM}}
repo2 {@{Branch=branch1; Date=2/6/2023 11:53:33 AM}, @{Branch=branch2; Date=2/6/2023 11:53:33 AM}, @{Branch=branch3; Date=2/6/2023 11:53:33 AM}}
repo3 {@{Branch=branch1; Date=2/6/2023 11:53:33 AM}, @{Branch=branch2; Date=2/6/2023 11:53:33 AM}, @{Branch=branch3; Date=2/6/2023 11:53:33 AM}}
repo4 {@{Branch=branch1; Date=2/6/2023 11:53:33 AM}, @{Branch=branch2; Date=2/6/2023 11:53:33 AM}, @{Branch=branch3; Date=2/6/2023 11:53:33 AM}}
repo5 {@{Branch=branch1; Date=2/6/2023 11:53:33 AM}, @{Branch=branch2; Date=2/6/2023 11:53:33 AM}, @{Branch=branch3; Date=2/6/2023 11:53:33 AM}}

This isn't the same as returning a dictionary keyed by branch names, but it isn't hard to adapt the solution to do that:

$dict = @{}  # initialize output dictionary (hashtable)
Get-StaleBranches | 
  ForEach-Object {
     # Create an entry for the repo at hand.
     $dict[$_.Repo] = $_.Results
  }

# Output the resulting dictionary.
$dict

Display output (the Name column contains the dictionary's key values):

Name     Value
----     -----
repo4    {@{Branch=branch1; Date=2/6/2023 12:01:42 PM}, @{Branch=branch2; Date=2/6/2023 12:01:42 PM}, @{Branch=branch3; Date=2/6/2023 12:01:42 PM}}
repo2    {@{Branch=branch1; Date=2/6/2023 12:01:42 PM}, @{Branch=branch2; Date=2/6/2023 12:01:42 PM}, @{Branch=branch3; Date=2/6/2023 12:01:42 PM}}
repo5    {@{Branch=branch1; Date=2/6/2023 12:01:42 PM}, @{Branch=branch2; Date=2/6/2023 12:01:42 PM}, @{Branch=branch3; Date=2/6/2023 12:01:42 PM}}
repo3    {@{Branch=branch1; Date=2/6/2023 12:01:42 PM}, @{Branch=branch2; Date=2/6/2023 12:01:42 PM}, @{Branch=branch3; Date=2/6/2023 12:01:42 PM}}
repo1    {@{Branch=branch1; Date=2/6/2023 12:01:42 PM}, @{Branch=branch2; Date=2/6/2023 12:01:42 PM}, @{Branch=branch3; Date=2/6/2023 12:01:42 PM}}

Thank you so much, the suggestion from the comment already helped immensely in speeding up the script, I will try this approach tomorrow morning. — Thyica, Feb 06 '23 at 17:00

How to add foreach-object -parallel output to a dictionary/arraylist

1 Answers1