0

I have a script that sort and zip Files in an folder by Year/Month. In normal folders its works fine with no errors but in folders with million files I get the error message:

System.OutOfMemoryException

Code with the Problem:

$files = Get-ChildItem $targetPath -Exclude "*.zip" -Recurse  | where-Object {!$_.PsIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-15)}

The files in the folder are really small together they have like 170MB so so its just the mass of the files that creates this problem.

Full Code:

$targetPath = 'C:\FolderWithMillionOfFiles'

$7zipPath = "C:\Program Files\7-Zip\7z.exe"
Set-Alias 7zip $7zipPath


#Select files to sort/filter
$files = Get-ChildItem $targetPath -Exclude "*.zip" -Recurse  | where-Object {!$_.PsIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-15)}


$files

foreach ($file in $files)
{
#get the year/month of a file
$year = $file.LastWriteTime.Year.ToString()
$month = $file.LastWriteTime.Month.ToString()

#Out FileName, year and month
$file.Name
$year
$month
 
#Folder thats getting zipped later
$Directory = $targetPath+ "\" +$year + "-" + $month 


if (!(Test-Path $Directory))
{
New-Item $directory -type directory #-set LastWriteTime=(Get-Date).AddDays(-15)
}

#Transfer the file to the folder with the year/time
$file | Move-Item -Destination $Directory
}

<#Works in power shell 5.1 but not in 4 because of the compress syntax
$filesneu = Get-ChildItem $targetPath -Exclude "*.zip" 
foreach ($file in $filesneu)
{Compress-Archive -Path $file -Update -Destination $file 
} #>

#works in powershell 4.0
$filesneu = Get-ChildItem $targetPath -Exclude "*.zip" 
foreach ($file in $filesneu){
7zip -tzip a -mx=9 $file $file
}

#Is deleting the folders so only the zipped files and files that are younger thatn 15 days stay in the folder (dont work right now because the folders are too young too so if u have a solution for this would be nice too
#Remove-item $targetPath\* -recurse -Exclude "*.zip" -force | where-Object {!$_.PsIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-15)} 

I already have maxed out the MaxMemoryPerShellMB and expected that this should solve the Problem (System.OutOfMemoryException in long running script)

I also tried a batch to split up the big folder in subfolders with like 100k files each (Fast methods to copy(move) files in batch file) but it works only with small folders like the script.

I tried the suggestions of the comments and my edited script part looks like this:

$targetPath = 'C:\SourcePath\Folderwithmillionfiles'

$7zipPath = "C:\Program Files\7-Zip\7z.exe"
Set-Alias 7zip $7zipPath

Get-ChildItem $targetPath -file -Exclude "*.zip" -Recurse | where-Object {$_.LastWriteTime -lt (Get-Date).AddDays(-15)} 

|ForEach-Object {Write-Host $_}

Sadly the script doesn't work like this and I get the error message:

an empty pipe element is not allowed

It seems the pipeline to the ForEach is empty but have no clue why.

phuclv
  • 37,963
  • 15
  • 156
  • 475
  • 2
    Have you tried to use the pipeline to keep memory usage down, instead of loading the entire list into memory and processing it. Something like `Get-ChildItem $targetPath -Exclude "*.zip" -Recurse | where-Object {!$_.PsIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-15)} | % {Rest of the code block} ` – Vivek Kumar Singh Nov 18 '22 at 07:52
  • I tried your pipeline operator but the script seems not to be working with this :/ Its finishes in an instant and didnt move files or creates folders to zip. Seems like it didnt get the $files like this :/ – Informatik Nov 18 '22 at 09:06
  • Which line is giving error? It may be the version of Zip that you are using that can't handle huge files or large number of files. – jdweng Nov 18 '22 at 09:34
  • 2
    Please read [about Pipelines](https://learn.microsoft.com/powershell/module/microsoft.powershell.core/about/about_pipelines). Don't assign anything (or use parenthesis), just pipes (`|`): `Get-ChildItem $targetPath -Exclude "*.zip" -Recurse | where-Object {!$.PsIsContainer -and $.LastWriteTime -lt (Get-Date).AddDays(-15)} |ForEach-Object { Write-Host 'Current Item:' $_; ...; ...; Remove-item $targetPath\* -recurse -Exclude "*.zip" }` – iRon Nov 18 '22 at 10:57
  • 2
    On a side note, remove `!$_.PsIsContainer` in the `Where-Object` script and use `Get-ChildItem -File` instead, which improves the speed as directories will be filtered by `Get-ChildItem` already. It still recurses into sub directories, but it will only output file entries. – zett42 Nov 18 '22 at 11:11
  • Having millions of files in a folder is a very very very bad idea regardless of FS & OS [performance implications for millions of files in a modern FS](https://serverfault.com/q/796665/343888) https://serverfault.com/q/43133 https://superuser.com/q/1733104/241386 https://unix.stackexchange.com/q/630611/44425 and it's **especially bad with NTFS** [Can file system performance decrease if there is a very large number of files in a single directory?](https://superuser.com/q/623965/241386), [Performance associated with storing millions of files on NTFS](https://serverfault.com/q/622872/343888) – phuclv Nov 23 '22 at 10:10
  • possible duplicate: [Windows directory with 5 million files](https://stackoverflow.com/q/53781139/995714), [Getting files in a directory that has over 7 million items using powershell](https://stackoverflow.com/q/67528750/995714) – phuclv Nov 23 '22 at 10:12
  • Does this answer your question? [Getting files in a directory that has over 7 million items using powershell](https://stackoverflow.com/questions/67528750/getting-files-in-a-directory-that-has-over-7-million-items-using-powershell) – phuclv Nov 23 '22 at 10:12
  • Acutally the remove of the !$_.PsIsContainer got it for me. Sadly i tested the Solutions together and that didnt work.. So right now the Skript is working slowly but doing its Job. Thanks for all the different inputs! If u have some tipps to do boost the scripts speed I would be really glad! (thought about chaning the move command to a robocopy command.. – Informatik Nov 23 '22 at 12:28

0 Answers0