0

I have a directory with over 5 million small files on windows. What's the best way to take out batches of files from that directory.

So far I've tried :powershell even simple gci | select -First 1 hangs forever :robocopy with a number of different parameters - its not able to move a file either.

Is there a way to grab x number of files without windows listing the entire directory in the process?

Thank you

VerdelothRahl
  • 21
  • 3
  • 1
    By `take out` you mean destroy? Please show what you have tried already, both with PowerShell and Robocopy . – Theo Dec 14 '18 at 13:59
  • No by take out i mean move them out to another directory. – VerdelothRahl Dec 14 '18 at 14:00
  • 9
    Try `[System.IO.Directory]::EnumerateFiles(dir) | Select -First 1`. This is a fairly simple wrapper around `FindFirstFile`, which doesn't attempt to do anything fancy with ordering the entries. If *that* takes too long, your only recourse is probably to wait. – Jeroen Mostert Dec 14 '18 at 14:06
  • What's wrong with a simple `xcopy`? Adding `/F` you can even get a complete log of all files which have been copied. – Dominique Dec 14 '18 at 14:14
  • Jeroen Mostert that worked great, thank you. – VerdelothRahl Dec 14 '18 at 14:18
  • Does this answer your question? [Getting files in a directory that has over 7 million items using powershell](https://stackoverflow.com/questions/67528750/getting-files-in-a-directory-that-has-over-7-million-items-using-powershell) – phuclv Nov 23 '22 at 10:17

1 Answers1

0

This solved the problem. Credit goes to Jeroen Mostert

Try [System.IO.Directory]::EnumerateFiles(dir) | Select -First 1. This is a fairly simple wrapper around FindFirstFile, which doesn't attempt to do anything fancy with ordering the entries. If that takes too long, your only recourse is probably to wait. – Jeroen Mostert

VerdelothRahl
  • 21
  • 3
  • You can even extend this using Linq `Take` method to return batches of filenames in an array like this: `$batchsize = 50; $files = ([System.Linq.Enumerable]::Take([System.IO.Directory]::EnumerateFiles($pwd, "*.*", "AllDirectories"), $batchsize)) | ForEach-Object { $_.ToString() }`. Instead of `$pwd` you can give it any root path you want ofcourse. The `AllDirectories` makes it recursive through subfolders. If you don't want that, use `TopDirectoryOnly` (which is the default) – Theo Dec 14 '18 at 16:37
  • @Theo: there's no real need for LINQ here; `Select -First 50` has the same effect. – Jeroen Mostert Dec 14 '18 at 19:02