C# get only X number of files

Question

I want to process only X number of files at a time for below infinite loop. Below codes give me all files at a time, how to get only X number of files?

while (true)
{
    var files = new DirectoryInfo(@"path")
                .GetFiles()
                .OrderBy(p => p.LastWriteTimeUtc)
                .ToList();

    foreach (var f in files)
    {
        //do some processing
        Console.WriteLine(f.Name);

        f.Delete();
    }

    Thread.Sleep(5000);
}

Maybe you need a structure wehere you could assign a sequential integer to each file name, then you can get the names in batches like 1 to x, x+1, 2x-1, etc. An 2D array may work for that. — NoChance, Sep 01 '20 at 06:27
Maybe this is more sophesticated: https://stackoverflow.com/questions/15414347/how-to-loop-through-ienumerable-in-batches — NoChance, Sep 01 '20 at 06:29

Prasad Telkikar · Accepted Answer · 2022-07-25T09:34:51.570

You can use combination of Skip() .Take(X) which will process your files in batches

Skip() : Bypasses a specified number of elements in a sequence and then returns the remaining elements.

Take() : Returns a specified number of contiguous elements from the start of a sequence.

Process files in batches,

var batchSize = 10;   //Decide batch size I considered 10 files in a batch
var skipBatch = 0     //Skip count
while (true)
{
    var files = new DirectoryInfo(@"path").GetFiles()
       .OrderBy(p => p.LastWriteTimeUtc).ToList(); //store all files

    var skipIntervals = skipBatch * batchSize;
    //Exit condition from infinity loop
    if(skipIntervals > files.Count)
        break;

    var filesInBatch = files.Skip(skipIntervals).Take(batchSize);
    foreach (var f in filesInBatch)
    {
        //do some processing
        Console.WriteLine(f.Name);
        f.Delete();
    }
    Thread.Sleep(5000);
    skipBatch++;  //Increment skipBatch count by 1 as this batch is processed
}

Your method is totally fine until `Skip` will use a small number as pointed out in this [question](https://stackoverflow.com/questions/34531818/entity-framework-skip-take-is-very-slow-when-number-to-skip-is-big) `Skip` will become slow if use very large numbers — Stefano Cavion, Sep 01 '20 at 07:47
Not quite sure on your assumption, I had the same problem using `.Skip()` on an `Byte[]` for manipulating the data of an image received from a camera. I had a really big frame and `.Skip()` take a vey long time, i've solved looking at this [answer](https://stackoverflow.com/a/7359605/8835810) — Stefano Cavion, Sep 01 '20 at 08:24

score 1 · Answer 2 · answered Sep 01 '20 at 06:24

 while (true)
    {
        var files = new DirectoryInfo(@"path").GetFiles()
            .OrderBy(p => p.LastWriteTimeUtc).ToList();

        foreach (var f in files)
        {
           Thread.Sleep(5000);//should be up here
            //do some processing
            Console.WriteLine(f.Name);

            f.Delete();
        }

        
    }

The thread.sleep method is not in the right place

you can always use Await Task.Delay() much better than thread.sleep since it makes the whole program freeze up depending on what you are doing with it

I cannot see how this answers the question from OP. "how to get only X number of files" — Jonas W, Sep 01 '20 at 06:26

score 0 · Answer 3 · answered Sep 01 '20 at 06:25

Just keep a counter. Each time you process a file, increase it by one. When it hits the limit, you've ended the batch.

while (true)
{
    var counter = 0;
    var files = new DirectoryInfo(@"path").GetFiles()
        .OrderBy(p => p.LastWriteTimeUtc).ToList();
    foreach (var f in files)
    {
        //do some processing
        Console.WriteLine(f.Name);
        f.Delete();
        counter++;
        if (counter >= batchSize) 
        {
            Thread.Sleep(timeBetweenBatches);
            counter = 0;
        }
    }
    Thread.Sleep(5000);
}

C# get only X number of files

3 Answers3

Linked