How to free memory during long loops in powershell 7

Question

I wrote a script to go through a list of network shares and find strings in specific files, with multi-threads using ForEach-Object -Parallel in powershell 7.

The problem is that even if i use the garbage collector or remove-variable, after a few minutes the memory on my computer is near full.

Code :

$ErrorActionPreference= 'silentlycontinue'

$available_share_list = "share_list.txt"
function Iterate_over_share_t ()
{
   $unc = Get-Content $available_share_list
   $unc | foreach-Object -Parallel {
      $share=$_
      $share_safe = $share.replace("\","-")
      $share_safe = $share_safe.replace("$","")
      $share_safe = $share_safe.replace(":","")
      Get-ChildItem -Recurse $share -Depth 4 |
      ForEach-Object{
        function parseFile() {
            param(
                $ext
            )
            $extension=$ext[0]
            $unc=$ext[1]
            $content = Get-Content $unc
            $date = Get-Item $unc; $date_csv = $date.LastWriteTime.ToString("yyyy-MM-dd HH:mm")
            
            ## Here i write to files if string is found in $content
            ## And i then remove according variables
            Write-Output "string" | myfile.txt
            Remove-Variable -Name <var> -ErrorAction SilentlyContinue
            
            ## I then force the garbage collector
            if($found -eq $true){
                [System.GC]::Collect()
                [System.GC]::GetTotalMemory('forcefullcollection') | out-null
                [System.GC]::GetTotalMemory($true) | out-null
            }

        }

        $file = $_.FullName -replace '\r?\n\z'
        parseFile("txt",$file)
        
        Remove-Variable -Name $file -ErrorAction SilentlyContinue
      }
      Remove-Variable -Name $share -ErrorAction SilentlyContinue
      Remove-Variable -Name $share_safe -ErrorAction SilentlyContinue
    [System.GC]::Collect()
    [System.GC]::GetTotalMemory('forcefullcollection') | out-null
    [System.GC]::GetTotalMemory($true) | out-null
    echo "[v] : $share"
   } -ThrottleLimit 50
  }

So i was wondering if anyone experienced this issue, or if i should just change to another language ahah.

Have a nice day, Q.

If you want to preserve memory, you better [*correctly*](https://learn.microsoft.com/powershell/scripting/developer/cmdlet/strongly-encouraged-development-guidelines#support-well-defined-pipeline-input-sc02) use the [PowerShell pipeline](https://learn.microsoft.com/powershell/module/microsoft.powershell.core/about/about_pipelines), for a better understanding, see also: [Mastering the (steppable) pipeline](https://devblogs.microsoft.com/powershell-community/mastering-the-steppable-pipeline/) — iRon, Jul 07 '23 at 16:06
So if I read correctly the way I’m using the pipeline with the foreach object is costing a lot of ressources ? — Quentin_otd, Jul 07 '23 at 16:12
Yes, but I recommend you to first write an [*advanced* function](https://learn.microsoft.com/powershell/module/microsoft.powershell.core/about/about_functions_advanced) for [the Middle of a Pipeline](https://learn.microsoft.com/powershell/scripting/developer/cmdlet/strongly-encouraged-development-guidelines#support-well-defined-pipeline-input-sc02) — iRon, Jul 07 '23 at 16:26
Alright I will try that and let you know, but I’m curious, if I only used variables instead of pipelines, isn’t it better ? — Quentin_otd, Jul 07 '23 at 16:36
E.g.: the `parseFile()` function should go outside the loop and be invoked like this: `Get-ChildItem ... | parseFile | ...` — iRon, Jul 07 '23 at 16:36
Better for what? Memory use? Probably not, Performance? Maybe... anyways, the PowerShell pipeline is often underestimated especially in situations where it concerns a slow input (as network shares) it immediately starts processing. See also [Fastest Way to get a uniquely index item from the property of an array](https://stackoverflow.com/a/59437162/1701026) — iRon, Jul 07 '23 at 16:42
Alright I will use the function in the middle of the pipeline. Do I define the function inside or outside the current thread, if outside I will need to use $using — Quentin_otd, Jul 07 '23 at 16:44

How to free memory during long loops in powershell 7

0 Answers0