Flatten directory structure

Question

The below function flattens the directory structure and copies files based on the last write date chosen.

function mega-copy($srcdir,$destdir,$startdate,$enddate)
{
    $files = Get-ChildItem $SrcDir -recurse | Where-Object { $_.LastWriteTime -ge "$startdate" -and $_.LastWriteTime -le "$enddate" -and $_.PSIsContainer -eq $false };
    $files|foreach($_)
    {
        cp $_.Fullname ($destdir+$_.name) -Verbose
    }
}

This has been very successful on smaller directories, but when attempting to use it for directories with multiple sub-directories and file counts ranging from the hundreds of thousands to the tens of millions, it simply stalls. I ran this and allowed it to sit for 24 hours, and not a single file was copied, nor did anything show up in the PowerShell console window. In this particular instance, there were roughly 27 million files.

However a simplistic batch file did the job with no issue whatsoever, though it was very slow.

Don't use the intermediate variable: http://stackoverflow.com/a/39657987/3959875 — wOxxOm, Sep 27 '16 at 16:19
This is something that robocopy could be better suited for and like wOxxOm says don't save the file in a variable. Use the pipeline. If you have at least PowerShell 3.0 you can also use the `-File` switch of Get-Childitem. — Matt, Sep 27 '16 at 16:45
Nice. Both of you provided me information that was relevant and seemed to decrease the amount of time it will take to move large amounts of data with this method. Thank you so much. — GreatMagusKyros, Sep 27 '16 at 17:40

score 2 · Accepted Answer · answered Sep 27 '16 at 19:06

Simple answer is this: using the intermediate variable caused a huge delay in the initiation of the file move. Couple that with using

-and $_.PSIsContainer -eq $false

as opposed to simply using the -file switch, and the answer was a few simple modifications to my script resulting in this:

function mega-copy($srcdir,$destdir,$startdate,$enddate)
{
Get-ChildItem $SrcDir -recurse -File | Where-Object { $_.LastWriteTime -ge "$startdate" -and $_.LastWriteTime -le "$enddate" } | foreach($_) {
                cp $_.Fullname ($destdir+$_.name) -Verbose
}
}

Flatten directory structure

1 Answers1

Linked