I have a script that sort and zip Files in an folder by Year/Month. In normal folders its works fine with no errors but in folders with million files I get the error message:
System.OutOfMemoryException
Code with the Problem:
$files = Get-ChildItem $targetPath -Exclude "*.zip" -Recurse | where-Object {!$_.PsIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-15)}
The files in the folder are really small together they have like 170MB so so its just the mass of the files that creates this problem.
Full Code:
$targetPath = 'C:\FolderWithMillionOfFiles'
$7zipPath = "C:\Program Files\7-Zip\7z.exe"
Set-Alias 7zip $7zipPath
#Select files to sort/filter
$files = Get-ChildItem $targetPath -Exclude "*.zip" -Recurse | where-Object {!$_.PsIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-15)}
$files
foreach ($file in $files)
{
#get the year/month of a file
$year = $file.LastWriteTime.Year.ToString()
$month = $file.LastWriteTime.Month.ToString()
#Out FileName, year and month
$file.Name
$year
$month
#Folder thats getting zipped later
$Directory = $targetPath+ "\" +$year + "-" + $month
if (!(Test-Path $Directory))
{
New-Item $directory -type directory #-set LastWriteTime=(Get-Date).AddDays(-15)
}
#Transfer the file to the folder with the year/time
$file | Move-Item -Destination $Directory
}
<#Works in power shell 5.1 but not in 4 because of the compress syntax
$filesneu = Get-ChildItem $targetPath -Exclude "*.zip"
foreach ($file in $filesneu)
{Compress-Archive -Path $file -Update -Destination $file
} #>
#works in powershell 4.0
$filesneu = Get-ChildItem $targetPath -Exclude "*.zip"
foreach ($file in $filesneu){
7zip -tzip a -mx=9 $file $file
}
#Is deleting the folders so only the zipped files and files that are younger thatn 15 days stay in the folder (dont work right now because the folders are too young too so if u have a solution for this would be nice too
#Remove-item $targetPath\* -recurse -Exclude "*.zip" -force | where-Object {!$_.PsIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-15)}
I already have maxed out the MaxMemoryPerShellMB
and expected that this should solve the Problem (System.OutOfMemoryException in long running script)
I also tried a batch to split up the big folder in subfolders with like 100k files each (Fast methods to copy(move) files in batch file) but it works only with small folders like the script.
I tried the suggestions of the comments and my edited script part looks like this:
$targetPath = 'C:\SourcePath\Folderwithmillionfiles'
$7zipPath = "C:\Program Files\7-Zip\7z.exe"
Set-Alias 7zip $7zipPath
Get-ChildItem $targetPath -file -Exclude "*.zip" -Recurse | where-Object {$_.LastWriteTime -lt (Get-Date).AddDays(-15)}
|ForEach-Object {Write-Host $_}
Sadly the script doesn't work like this and I get the error message:
an empty pipe element is not allowed
It seems the pipeline to the ForEach is empty but have no clue why.