Reducing the time needed to gather files from a larger amount of directories

Question

Hiho.

I'm not sure if it is possible to decrease loading times in my program, or if I just have to switch to SSDs. But maybe someone has an Idea :)

I'm getting the first (max) 100 files from a large number of folders and write their path to a single list. The problem is that it takes ages to load this list because I always get the whole content of the folder to a temporary list and then write the first 100 entries of that temporary list to my final list. Maybe it is faster if I could skip this extra step but I'm not sure if this is possible and if this would make any difference.

My code looks like this:

foreach (string Folder in FolderList)
{
    List<string> FileListTMP = new List<string>();
    FileListTMP.AddRange(Directory.GetFiles(Folder));
    int i = 0;
    while (i < FileListTMP.Count && i < 100)
    {
        FileList.Add(FileListTMP[i]);
        i++;
    }
}

It would be great if there is a solution to my problem because the number of folders is always growing and the loading times are already around 20 seconds long. :)

More info: At this time there are around 1500 folders. There are not more than 1000 files in a single folder. The average is about 500 files per folder.

You can use [EnumerateFiles](https://learn.microsoft.com/en-us/dotnet/api/system.io.directoryinfo.enumeratefiles) instead of `GetFiles`, like `FileList.AddRange(Directory.EnumerateFiles(Folder).Take(100))`. — Klaus Gütter, Jul 21 '23 at 12:02
How many files has each folder on average and at maximum? How many are the folders? — Theodor Zoulias, Jul 21 '23 at 12:22
As usual you should profile. Using EnumerateFiles might help but you should also check out the Antivirus overhead. See https://stackoverflow.com/questions/76012101/c-sharp-reading-sequentially-text-files-from-ssd-drive-is-very-slow-the-first-ti/76062486#76062486 for an example. You might be surprised where your actual bottleneck is. — Alois Kraus, Jul 21 '23 at 12:45
There are not more than 1000 files in a single folder. I guess the average is about 500 files per folder. At this time there are around 1500 folders. — xardius159, Jul 21 '23 at 12:48
Thank you all for the help. With using EnumerateFiles and .Take(100) I could decrease the loading time to under 8 seconds. I also included a logic that reduces the files I get from a single folder based on the amount of folders, with this the loading time is most of the time under 5 seconds (as far as i could verify this in the last hour). I'll test a bit more but it looks quite good now. :) — xardius159, Jul 21 '23 at 13:27
Do you have permission to write and rename these files? If so, I suggest creating a function that generates the path based on the file name, so you can index in ascending order and get that file on time. Example: `public string GetFullPathFile(string path, int fileName)` => ` GetFullPathFile("C:\\TestDir", 999)` => `C:\TestDir\1\1\1\999.txt` — Propeus, Jul 21 '23 at 16:03

score 3 · Accepted Answer · answered Jul 21 '23 at 12:04

Your posted code is directly equivalent to

FileList.AddRange(Directory.GetFiles(Folder).Take(100));

But the only thing that should take significant time in that piece of code is the GetFiles-call. Moving things between lists can be a problem if you are processing many millions of items, but should not be an issue when talking about hundreds of items. But you should use a performance profiler to verify this assumption, perhaps there is something completely different that takes time.

You could try changing .GetFiles to .EnumerateFiles since the later returns an enumerable rather than an array, so might be faster if you do not need a complete list of all files.

In my experience it might also be possible to get a performance speedup when listing files by using the native API rather than GetFiles. Something similar to this answer. But it does require a decent chunk of somewhat complex code.

Thank you. EnumerateFiles decreases the loading time from about 20 seconds to a bit over 15. It's better than nothing :) — xardius159, Jul 21 '23 at 12:51

Reducing the time needed to gather files from a larger amount of directories

1 Answers1