Powershell file parsing very slow

Question

I am using Powershell 7.

We have the following PowerShell script that will parse some very large file.

I no longer want to use 'Get-Content' as this is to slow.

The script below works, but it takes a very long time to process even a 10 MB file.

I have about 200 files 10MB file with over 10000 lines.

Sample Log:

#Fields:1
#Fields:2
#Fields:3
#Fields:4
#Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context
2023-01-31T13:53:50.404Z,EXCH1\Relay-EXCH1,08DAD23366676FF1,41,10.10.10.2:25,195.85.212.22:15650,<,DATA,
2023-01-31T13:53:50.404Z,EXCH1\Relay-EXCH1,08DAD23366676FF1,41,10.10.10.2:25,195.85.212.25:15650,<,DATA,

Script:

$Output = @()
$LogFilePath = "C:\LOGS\*.log"
$LogFiles = Get-Item  $LogFilePath
$Count = @($logfiles).count

ForEach ($Log in $LogFiles)
{
    $Int = $Int + 1
    
    $Percent = $Int/$Count * 100

    Write-Progress -Activity "Collecting Log details" -Status "Processing log File $Int of $Count - $LogFile" -PercentComplete $Percent 

    Write-Host "Processing Log File  $Log" -ForegroundColor Magenta
    Write-Host
    $FileContent = Get-Content $Log | Select-Object -Skip 5
    ForEach ($Line IN $FileContent)
    {

        $Socket = $Line  | Foreach {$_.split(",")[5] }

        $IP = $Socket.Split(":")[0]

        $Output += $IP

    } 
} 
$Output = $Output | Select-Object -Unique
$Output = $Output | Sort-Object

Write-Host "List of noted remove IPs:" 
$Output
Write-Host 
$Output | Out-File $PWD\Output.txt 

Answer 1

As @iRon Suggests the assignment operator (+=) is a lot of overhead. As well as reading entire file to a variable then processing it. Perhaps process it strictly as a pipeline. I achieved same results, using your sample data, with the code written this way below.

$LogFilePath = "C:\LOGS\*.log"
$LogFiles = Get-ChildItem $LogFilePath
$Count = @($logfiles).count

$Output = ForEach($Log in $Logfiles) {
    # Code for Write-Progress here
    Get-Content -Path $Log.FullName | Select-Object -Skip 5 | ForEach-Object {
        $Socket = $_.split(",")[5] 
        $IP = $Socket.Split(":")[0]
        $IP
    }
}
$Output = $Output | Select-Object -Unique
$Output = $Output | Sort-Object

Write-Host "List of noted remove IPs:" 
$Output

Answer 2

Apart from the notable points in the comments, I believe this question is more suitable to Code Review. Nonetheless, here's my take on this using the StreamReader class:

$LogFilePath = "C:\LOGS\*.log"
$LogFiles    = Get-Item -Path $LogFilePath
$OutPut      = [System.Collections.ArrayList]::new()

foreach ($log in $LogFiles)
{
    $skip = 0
    $stop = $false
    $stream = [System.IO.StreamReader]::new($log.FullName)
    while ($line = $stream.ReadLine())
    {
        if (-not$stop)
        {
            if ($skip++ -eq 5)
            {
                $stop = $true
            }
            continue
        }
        elseif ($OutPut.Contains(($IP = ($line -split ',|:')[6])))
        {
            continue
        }
        $null = $OutPut.Add($IP)
    }
    $stream.Close()
    $stream.Dispose()
}
# Display OutPut and save to file
Write-Host -Object "List of noted remove IPs:" 
$OutPut | Sort-Object | Tee-Object -FilePath "$PWD\Output.txt"

This way you can output unique IP's since it's being handled by an if statement checking against what's in $OutPut; essentially replacing Select-Object -Unique. You should see a speed increase as you're no longer adding to a fixed array (+=), and piping to other cmdlets.

Answer 3

You can combine File.ReadLines with Enumerable.Skip to read your files and skip their first 5 lines. This method is much faster than Get-Content. Then for sorting and getting unique strings at the same time you can use a SortedSet<T>.

You should avoid using Write-Progress as this will slow your script down in Windows PowerShell (this has been fixed in newer versions of PowerShell Core).

Do note that because you're looking to sort the result, all strings must be contained in memory before outputting to a file. This would be much more efficient if sorting was not needed, there you would use a HashSet<T> instead for getting unique values.

Get-Item C:\LOGS\*.log | & {
    begin { $set = [Collections.Generic.SortedSet[string]]::new() }
    process {
        foreach($line in [Linq.Enumerable]::Skip([IO.File]::ReadLines($_.FullName), 5)) {
            $null = $set.Add($line.Split(',')[5].Split(':')[0])
        }
    }
    end {
        $set
    }
} | Set-Content $PWD\Output.txt