Directory Enumeration Too Slow for First Time

Question

I want to try making own file explorer. I have a algorithm to enumerate all directories in all drives. But it runs far too slowly. Here is my code:

public ExplorerForm()
{
    InitializeComponent();
    this.SuspendLayout();//Without this, it will be far more slow again!
    string[] a = System.IO.Directory.GetLogicalDrives();// a is used for drive array
    for(int b = 0; b < a.Length; b++)//B is an enumerator. Ussually it is only one letter
    {
        //Defining the node
        TreeNode c = new TreeNode();//c i place for TreeNode
        c.Text = a[b].Substring(0,2);
        c.Tag = a[b];
        ApplyNodes(a[b], ref c);
        if(c != null) tv.Nodes.Add(a)
    }
    this.ResumeLayout(false);
}
private void ApplyNodes(string a, ref TreeNode b)//a=directory, b applied TreeNode
{
    try{
        List<string> c = new List<string>(Directory.EnumerateDirectories(a);//c = directories
        if (c.Count == 0 ) return;
        for(int d = 0; d < c.Count; d++)//d = enumerator.
        {
            TreeNode e = new TreeNode();//e = TreeNode
            var z = c[b].Split(Convert.ToChar("/"));
            e.Text = z[z.Length-1]
            e.Tag = c[b];
            ApplyNodes(c[b], e)
            if(e != null) b.Nodes.Add(e)
        }
    }catch (UnauthorizedAccessException){
    }catch (IOException){  //For test, It is removed. and my E: is not ready
    }
}

tv is my control. It is run very slowly. It is indicated when I remove choosen line, It will take time more than 10s to throw an IOException. Help me how to improve the enumeration. This is except using thread and partial update. If can't be later fixed, tell me why.

Err..yeah. That's why you don't enumerate all the directories. You just show the ones you need when they are requested. — aquinas, Jul 06 '13 at 15:57
Other file managers work with the same slow interface. Look closely how they operate. — H H, Jul 06 '13 at 16:02
@aquinas, you are right. But how to update. And, I think that is slower than Windows' FIle Explorer. — Christian Irwan Hadi Wicaksana, Jul 06 '13 at 23:40
*Help me how to improve the enumeration. This is except using thread and partial update.* You're basically saying "This is slow. Make it faster without the accepted techniques of making it faster." Do you want academic solutions to your problem, or practical ones? — ta.speot.is, Jul 07 '13 at 05:18
I know that it can be later upgrade using thread. But now I just want this. But I also know for first time, it will very slow, but later it will be faster. But, I don't want to be unbanned again. — Christian Irwan Hadi Wicaksana, Jul 07 '13 at 05:59
*No, Windows' File Explorer just need two seconds* Open up Windows Explorer and click on your `C:` drive on the left hand side in the folder browser. Push the `*` key on your numpad. Count to one mississipi and then two mississippi. Is it finished? No. Minimize the window if you want to reduce the effects of redrawing. — ta.speot.is, Jul 07 '13 at 07:43
Nothing to do with performance - but please consider more meaningful variable names than a, b, c and d. It makes your code unnecessarily hard to understand. — Neville Kuyt, Jul 07 '13 at 08:36
@NevilleK I was added the comment for each variable. Is it better now? — Christian Irwan Hadi Wicaksana, Jul 08 '13 at 01:44
Enumerating all files in all drives always takes a lot of time. If I do a search for a filename with windows explorer it takes several minutes. So why do you expect such an enumeration to be fast? — CodesInChaos, Jul 08 '13 at 08:10
I want to ask now, prevent the my "spam" answer again. How to test to produce right answer? — Christian Irwan Hadi Wicaksana, Jul 08 '13 at 14:06

score 5 · Answer 1 · edited Jun 20 '20 at 09:12

Aside from the time it takes to enumerate all the directories on your system (you'd likely see better performance if you implement lazy loading), inserting items into a TreeView can take a significant amount of time.

From TreeView.BeginUpdate:

To maintain performance while items are added one at a time to the TreeView, call the BeginUpdate method. The BeginUpdate method prevents the control from painting until the EndUpdate method is called. The preferred way to add items to a tree view control is to use the AddRange method to add an array of tree node items to a tree view.

...

To allow the control to resume painting, call the EndUpdate method when all the tree nodes have been added to the tree view.

Although it departs from .NET, Raymond Chen's blog post How to insert a large number of items into a treeview efficiently has more information that might help you structure your code in a way that leads to better item insertion performance.

If you need to insert a large number of items into a treeview, like tens of thousands, then it's much more efficient to insert them "backwards".

EDIT

Here's an example that puts the enumeration of directories onto a thread. Observe the usability of the TreeView control (or lack thereof). If nothing else, this is probably the best argument for using lazy loading.

private void Form1_Load(object sender, EventArgs e)
{
    var treeNode = new TreeNode("Sea Drive");
    treeView1.Nodes.Add(treeNode);

    ThreadPool.QueueUserWorkItem(_ => TraverseDirectory("C:\\", treeNode));
}
   
private static readonly string DirectorySeparatorString = Path.DirectorySeparatorChar.ToString();

private void TraverseDirectory(string initialDirectoryPath, TreeNode initialTreeNode)
{
    var initialTuples = new[] {Tuple.Create(initialDirectoryPath, initialTreeNode)};
    var directoryQueue = new Queue<Tuple<string, TreeNode>>(initialTuples);

    while (directoryQueue.Any())
    {
        var tuple = directoryQueue.Dequeue();
        var parentDirectoryPath = tuple.Item1;
        var parentTreeNode = tuple.Item2;

        try
        {
            var treeNodes = new List<TreeNode>();
            var directories = Directory.EnumerateDirectories(parentDirectoryPath);

            foreach (var directoryPath in directories)
            {
                var lastDirectorySeparator = directoryPath.LastIndexOf(DirectorySeparatorString);
                var directoryName = directoryPath.Substring(lastDirectorySeparator + 1);

                // Add the tree node to our list of child 
                // nodes, for an eventual call to AddRange
                var treeNode = new TreeNode(directoryName);
                treeNodes.Add(treeNode);

                // We have to go deeper
                directoryQueue.Enqueue(Tuple.Create(directoryPath, treeNode));
            }

            // Run this operation on the main thread
            Invoke((Action)(() => parentTreeNode.Nodes.AddRange(treeNodes.ToArray())));
        }
        catch (Exception exception)
        {
            Trace.Write(exception);
        }
    }
}

The example is not complete; you'll need to supply your own Form and TreeView control.

is the TreeView.BeginUpdate() is different than Form.SuspendLayout? — Christian Irwan Hadi Wicaksana, Jul 07 '13 at 05:25
Why don't we [read the fine manual](http://msdn.microsoft.com/en-AU/library/system.windows.forms.control.suspendlayout(v=vs.110).aspx) to find out? *`Control.SuspendLayout` Temporarily suspends the layout logic for the control* vs. *`TreeView.BeginUpdate` Disables any redrawing of the tree view*. — ta.speot.is, Jul 07 '13 at 05:26
It will helps. But from test, the real problem isn't from TreeView. It's also help if you sent me other way to enumerate with less severe slowdown at start. — Christian Irwan Hadi Wicaksana, Jul 07 '13 at 07:28
Click on the first link about lazy loading... *with the WinForms `TreeView` you need to have at least one child node or it won't show the expand [+], but then you handle the `TreeNodeExpanded` event to remove that dummy node and populate the children*. Also see my comment [here](http://stackoverflow.com/questions/17504522/directory-enumeration-too-slow-for-first-time#comment25455908_17504522). It seems you already know solutions to solve this problem, you're just discarding them for no good reason. — ta.speot.is, Jul 07 '13 at 07:37
@ta.speot.is So, there aren't any solution except to implement in the thread and partial update? — Christian Irwan Hadi Wicaksana, Jul 08 '13 at 01:57

holtavolt · Answer 2 · 2013-07-07T08:27:53.937

In addition to the prior answer about improving your TreeView population calls, you should read the MSDN page How to: Enumerate Directories and Files

The first paragraph mentions some performance improvements (using enumerable collections of DirectoryInfo instead of strings) - notice the last line:

You can enumerate directories and files by using methods that return an enumerable collection of strings of their names. You can also use methods that return an enumerable collection of DirectoryInfo, FileInfo, or FileSystemInfo objects. Enumerable collections provide better performance than arrays when you work with large collections of directories and files.

Even with this improvement, however, you really should not recursively descend the entire subtree inside of ApplyNodes. Just read a single level, adding the entries for your current node, to greatly reduce the number of sub-directories you need to traverse (this is certainly something that File Explorer does.) This is the point of the "lazy loading" technique mentioned above by ta.speot.is

If these two improvements still don't give you the performance you want, then you may want to add more complexity(e.g. running a background thread to perform your traversal), but you'll want to have a good idea of exactly what part of your code is your bottleneck first (meaning you'll want to add timing code and logging)

Using foreach? I was trying that. But the bottleneck is the `Directory.EnumerateDirectories()` — Christian Irwan Hadi Wicaksana, Jul 08 '13 at 01:47
You are using static methods of the Directory class. Try using the DirectoryInfo class instead for the enumeration, e.g. http://stackoverflow.com/questions/6239544/c-sharp-how-to-populate-treeview-with-file-system-directory-structure This may help some, but your best bet is to remove the fully recursive traversal of the directory subtree, and only traverse as far as needed for the current TreeView state (i.e. only traverse another level of a subtree in response to a user expanding a node) — holtavolt, Jul 08 '13 at 02:03

Directory Enumeration Too Slow for First Time

2 Answers2