How can i write logs to single file using foreach-object parallel?

Question

Issue

I am trying to write logs to a single log file using the PSlogging PowerShell module.

The Script is using the Foreach-object parallel.

The script works, But sometimes the script gets stuck in between.

Please help me to resolve it? Is there a better way to achieve this?

Powershell core 7

code

#Script Version
$sScriptVersion = '1.0'

$sScriptVersion = '1.0'
$sLogName = 'custom.log'
#Log File Info
$sLogPath = "C:\Users\Dixon\Desktop\automation\azure\log"
$sLogFile = Join-Path -Path $sLogPath -ChildPath $sLogName

Start-Log -LogPath  $sLogPath -LogName $sLogName  -ScriptVersion $sScriptVersion

$allServers = $(Get-AzVM -Status) 

try {
    $allServers | ForEach-Object -parallel {

        $sScriptVersion = '1.0'
        $sLogName = "parallel-" + $_.Name +".log"
        #Log File Info
        $sLogPath = "C:\Users\Dixon\Desktop\automation\azure\log"
        $sLogFile = Join-Path -Path $sLogPath -ChildPath $sLogName

        Import-Module .\modules\pwshAzMod\pwshAzMod.psd1 -Force

        $version = "1.0.11"
       
        
        $vmName = $_.Name
        $vmLocation = $_.Location
        $vmResourceGroupName = $_.ResourceGroupName
        $powerState = $_.PowerState | Out-String
        if ($powerState -match "running") {
            if ($_.OSProfile.WindowsConfiguration) {
                Write-LogInfo -LogPath $sLogFile -Message  "Start --- Windows Servers --- $vmResourceGroupName $vmLocation $vmName $version $powerState"
                Write-LogInfo -LogPath $sLogFile -Message  "Done --- Windows Servers --- $vmResourceGroupName $vmLocation $vmName $version"
            } 
            elseif ($_.OSProfile.LinuxConfiguration) {
 
                $extensionName = "CustomScript"
                Write-LogInfo -LogPath $sLogFile -Message  "Start --- Linux Servers --- $vmResourceGroupName $vmLocation $vmName $version $powerState"
                Write-LogInfo -LogPath $sLogFile -Message  "Done --- Linux Servers --- $vmResourceGroupName $vmLocation $vmName $version"
            }
            else {
                Write-LogInfo -LogPath $sLogFile -Message  "OS Type not handled"
            }
        }
        else {
            if ($_.OSProfile.WindowsConfiguration) {
                Write-LogInfo -LogPath $sLogFile -Message  "Deallocated --- Windows Servers --- $vmResourceGroupName $vmLocation $vmName $version $powerState"
            }
            elseif ($_.OSProfile.LinuxConfiguration) {
                Write-LogInfo -LogPath $sLogFile -Message  "Deallocated --- Linux Servers --- $vmResourceGroupName $vmLocation $vmName $version $powerState"
            }
            else {
                Write-LogInfo -LogPath $sLogFile -Message "Deallocated --- Other Servers --- $vmResourceGroupName $vmLocation $vmName $version $powerState"
            }
        }
    }  -ThrottleLimit 5
}
catch {
    $errorMessage = $_
    $errorMessage
}
finally {
    Get-Content $sLogPath/parallel-*.log | Add-Content $sLogFile
    Remove-Item –path $sLogPath\*  -Filter parallel*
    Stop-Log -LogPath $sLogFile
}

As an aside: You don't need `$(...)` to capture a command's output in a variable - `$allServers = Get-AzVM -Status` will do. — mklement0, Mar 21 '20 at 20:52
Presumably, `Write-LogInfo`,from the [`PSLogging` module](https://www.powershellgallery.com/packages/PSLogging), wasn't designed with concurrency in mind, which means that you would need to use explicit synchronization between the threads. — mklement0, Mar 21 '20 at 20:54
One option is to place extra logic for explicit synchronization around the `Write-LogInfo` calls, as discussed in [this answer](https://stackoverflow.com/a/53805653/45375). — mklement0, Mar 21 '20 at 21:56

Dragoon · Answer 1 · 2020-10-02T20:24:36.227

2

I stumbled upon this via google and didn't like any of the solutions I found, so I made up my own for logging that you might want to try. This example is how to write to a text file in the current directory safely in a foreach parallel loop:

$tw = [System.IO.TextWriter]::Synchronized([System.IO.File]::AppendText("$PSScriptRoot\log.txt"))

0..100 | Foreach-Object -Parallel {
  $writer = $Using:tw
  $writer.WriteLine("hello $_")
}
$tw.Close()

Optionally wrap the whole thing in a try/finally with the $tw.Close() in the finally block to ensure that the log file buffer gets flushed and then closed if the script is interrupted.

The TextWriter object handles the locking for you and works just like a StreamWriter object. Note that this is assuming you aren't trying to also read from the file in any of the threads, for that you'd need something a lot more complex.

edited Oct 02 '20 at 20:24

answered Oct 02 '20 at 20:18

Dragoon

723
6
13

I'm not going to do the Stack Exchange expected of telling you off "for not answering what they asked" because I think this is one better alternative to the method they wanted. However the other user's answer better answers how to make it work with their existing methodology. I think there may another solution between yours and theirs which might be more apropos, but, in all I tend to agree with your solution more than fixing their solution, so good on you, I wanted to give you a note on this before upvoting as I was tasked with review on this new answer to an old question. – Ben Personick Oct 02 '20 at 21:04
1

Right the main reason I posted it here is because this is the first question I landed on when I used Google, so it's more likely that somebody else trying to do the same thing would want to try this approach. I could have posted it in the other solution I found for writing to an XML file that another commenter mentioned, but knowing how easy this is to do in .net I was only missing one piece of info, which was that $Using: bit to keep the $tw in scope for writing. This probably isn't an idiomatic way to do this in powershell like the XML solution, but it's short and does the trick. – Dragoon Oct 02 '20 at 21:11
I often prefer someone writing out a specific answer to the given question, over linking even if the answer is similar to another question's answers, for that very reason. People already googled they want answer to the question they asked. – Ben Personick Oct 02 '20 at 21:17

score 1 · Accepted Answer · answered Oct 02 '20 at 20:30

1

It won't be in order unless you sort it:

1..10 | foreach-object -Parallel { sleep 1;$_ } | set-content file.txt

answered Oct 02 '20 at 20:30

js2010

23,033
6
64
66

How can i write logs to single file using foreach-object parallel?

Issue

code

2 Answers2