How to access DirectoryInfo.EnumerateFiles before it has completed

Question

In the question I asked Retrieve a list of filenames in folder and all subfolders quickly And a few others I have found, it seems the way to search many files is to use the EnumerateFiles Method.

The EnumerateFiles and GetFiles methods differ as follows: When you use EnumerateFiles, you can start enumerating the collection of names before the whole collection is returned; when you use GetFiles, you must wait for the whole array of names to be returned before you can access the array. Therefore, when you are working with many files and directories, EnumerateFiles can be more efficient.

This sounds Great for me, my search is taking about 10 seconds, so I can start makign my list as the information comes in. But I can't figure it out. When I run the EnumerateFiles Method, the application freezes until it completes. I could run it in a background worker, but the same thing will happen to that thread. Any help?

 DirectoryInfo dir = new DirectoryInfo(MainFolder);
 List<FileInfo> matches = new List<FileInfo>(dir.EnumerateFiles("*.docx",SearchOption.AllDirectories));

//This wont fire until after the entire collection is complete
DoSoemthingWhileWaiting();

Answer 1

You can do this by pushing it into a background task.

For example, you could do:

var fileTask = Task.Factory.StartNew( () =>
{
    DirectoryInfo dir = new DirectoryInfo(MainFolder);
    return new List<FileInfo>(
           dir.EnumerateFiles("*.docx",SearchOption.AllDirectories)
           .Take(200) // In previous question, you mentioned only wanting 200 items
       );
};

// To process items:
fileTask.ContinueWith( t =>
{
     List<FileInfo> files = t.Result;

     // Use the results...
     foreach(var file in files)
     {
         this.listBox.Add(file); // Whatever you want here...
     }
}, TaskScheduler.FromCurrentSynchronizationContext()); // Make sure this runs on the UI thread

DoSomethingWhileWaiting();

---

You mentioned in a comment:

I want to display them in a list. and perfect send them to the main ui as they come in

In this case, you'd have to process them in the background, and add them to the list as they come in. Something like:

Task.Factory.StartNew( () =>
{
    DirectoryInfo dir = new DirectoryInfo(MainFolder);
    foreach(var tmp in dir.EnumerateFiles("*.docx",SearchOption.AllDirectories).Take(200))
    {
        string file = tmp; // Handle closure issue

        // You may want to do this in batches of >1 item...
        this.BeginInvoke( new Action(() =>
        {
             this.listBox.Add(file);
        }));
    }
});
DoSomethingWhileWaiting();

Answer 2

The application freezes because you are consuming this enumerable by putting it in the constructor of the List<FileInfo>. It's as if you was calling eagerly the old method. So the proper way to do this is to run it inside a background thread and then do not immediately pass the result to something that will consume the enumerable till the end but start iterating and add items as they arrive in the loop.

I could run it in a background worker, but the same thing will happen to that thread.

Yes, during the loop you will obviously freeze the background thread but that's what background threads are meant for: to avoid freezing the main UI thread. Just make sure that when you are passing the items to the main thread to show them you are using proper synchronization with the UI. In WinForms this happens with the Control.Invoke method. Of course be careful because marshalling quite often between the background thread and the UI thread could also have negative impact give you the feeling that your application freezes. To workaround this you could pass the files as they arrive in chunks.

Here's an example:

Task.Factory.StartNew(() =>
{
    var dir = new DirectoryInfo(MainFolder);
    var files = dir.EnumerateFiles("*.docx", SearchOption.AllDirectories);
    foreach (var file in files)
    {
        Action<string> del = f => listBox1.Items.Add((string)f);
        BeginInvoke(del, file);
    }
});

Answer 3

you can do

    forach(var f in dir.EnumerateFiles("*.docx",SearchOption.AllDirectories))
    {
//.. process file
    }

in thread, and fire an event after process complete