Parallel Loops for Folder Size Determination

Question

The overall programs goal is to determine the size of main folders in directories. It works pretty well for small drives, but struggles for larger ones. It took over 3 hours for one of the drives that I absolutely need. This is a copy of my folder sizing program, I am using.

    public  double getDirectorySize(string p)
    {

        //get array of all file names
        string[] a = Directory.GetFiles(p, "*.*", SearchOption.AllDirectories);

        //calculate total bytes in loop
        double b = 0;
        foreach (string name in a)
        {

            if (name.Length < 250) // prevents path too long errors
            {


                    //use file info to get length of each file 
                    FileInfo info = new FileInfo(name);
                    b += info.Length;
            }
        }

        //return total size
        return b;
    }

So what I was thinking of using parallel loops in a form of parallel foreach loops. Each p represents the main folder's name. I was thinking of somehow splitting path p into its subfolders and using parallel foreach loops to continue collecting file sizes; however, they have an unknown amount of subdirectories. This is where I am having problems with trying to get the folder size back. Thanks for the help in advance

Update

I call this function through this foreach loop below

           DirectoryInfo di = new DirectoryInfo    (Browse_Folders_Text_Box.Text);
            FileInfo[] parsedfilename = di.GetFiles("*.*", System.IO.SearchOption.TopDirectoryOnly);
            parsedfoldername = System.IO.Directory.GetDirectories(Browse_Folders_Text_Box.Text, "*.*", System.IO.SearchOption.TopDirectoryOnly);
            //parsedfilename = System.IO.Directory.GetDirectories(textBox1.Text, "*.*", System.IO.SearchOption.AllDirectories);





            // Process the list of folders found in the directory.

            type_label.Text = "Folder Names \n";


            List<string> NameList = new List<string>();
            foreach (string transfer2 in parsedfoldername)
            {

                this.Cursor = Cursors.WaitCursor;
                //Uses the path and takes the name from last folder used
                string dirName = new DirectoryInfo(@transfer2).Name;
                string dirDate = new DirectoryInfo(@transfer2).LastWriteTime.ToString();


                NameList.Add(dirName);
                //Form2 TextTable = new Form2(NameList.ToString());



                //Display_Rich_Text_Box.AppendText(dirName);
                //Display_Rich_Text_Box.AppendText("\n");
                Last_Date_Modified_Text_Box.AppendText(dirDate);
                Last_Date_Modified_Text_Box.AppendText("\n");


                try
                {
                    double b;

                    b = getDirectorySize(transfer2);
                    MetricByte(b);



                }
                catch (Exception)
                {
                    Size_Text_Box.AppendText("N/A \n");                      
                }

            }

            Display_Rich_Text_Box.Text = string.Join(Environment.NewLine, NameList);
            this.Cursor = Cursors.Default;

So what I was thinking when I thought of parallel foreach loops was to take the next instance names (subfolder name) that would be all on the same level and run them all at the same time with getDirectorySize() because I know there is at least 7 subfolders directly beneath the main folder name.

Answer 1

Parallel access to the same physical drive will not speed up the work.

Your main problem is the GetFiles method. It goes through all the subfolders collecting all file names. Then you pass in a loop on same files again.

Use the EnumerateFiles method instead.

Try this code. It will be much faster.

public long GetDirectorySize(string path)
{
    var dirInfo = new DirectoryInfo(path);
    long totalSize = 0;

    foreach (var fileInfo in dirInfo.EnumerateFiles("*.*", SearchOption.AllDirectories))
    {
        totalSize += fileInfo.Length;
    }
    return totalSize;
}

MSDN:

The EnumerateFiles and GetFiles methods differ as follows: When you use EnumerateFiles, you can start enumerating the collection of names before the whole collection is returned; when you use GetFiles, you must wait for the whole array of names to be returned before you can access the array. Therefore, when you are working with many files and directories, EnumerateFiles can be more efficient.

Answer 2

I've had to do something similar, though not for folder / file sizes.

I don't have the code handy, but I used the following as a starter. It executes in parallel if there are enough files in the directory

From the source on MSDN:

The following example iterates the directories sequentially, but processes the files in parallel. This is probably the best approach when you have a large file-to-directory ratio. It is also possible to parallelize the directory iteration, and access each file sequentially. It is probably not efficient to parallelize both loops unless you are specifically targeting a machine with a large number of processors. However, as in all cases, you should test your application thoroughly to determine the best approach.

   static void Main()
   {            
      try 
      {
         TraverseTreeParallelForEach(@"C:\Program Files", (f) =>
         {
            // Exceptions are no-ops.
            try {
               // Do nothing with the data except read it.
               byte[] data = File.ReadAllBytes(f);
            }
            catch (FileNotFoundException) {}
            catch (IOException) {}
            catch (UnauthorizedAccessException) {}
            catch (SecurityException) {}
            // Display the filename.
            Console.WriteLine(f);
         });
      }
      catch (ArgumentException) {
         Console.WriteLine(@"The directory 'C:\Program Files' does not exist.");
      }   

      // Keep the console window open.
      Console.ReadKey();
   }

   public static void TraverseTreeParallelForEach(string root, Action<string> action)
   {
      //Count of files traversed and timer for diagnostic output
      int fileCount = 0;
      var sw = Stopwatch.StartNew();

      // Determine whether to parallelize file processing on each folder based on processor count.
      int procCount = System.Environment.ProcessorCount;

      // Data structure to hold names of subfolders to be examined for files.
      Stack<string> dirs = new Stack<string>();

      if (!Directory.Exists(root)) {
             throw new ArgumentException();
      }
      dirs.Push(root);

      while (dirs.Count > 0) {
         string currentDir = dirs.Pop();
         string[] subDirs = {};
         string[] files = {};

         try {
            subDirs = Directory.GetDirectories(currentDir);
         }
         // Thrown if we do not have discovery permission on the directory.
         catch (UnauthorizedAccessException e) {
            Console.WriteLine(e.Message);
            continue;
         }
         // Thrown if another process has deleted the directory after we retrieved its name.
         catch (DirectoryNotFoundException e) {
            Console.WriteLine(e.Message);
            continue;
         }

         try {
            files = Directory.GetFiles(currentDir);
         }
         catch (UnauthorizedAccessException e) {
            Console.WriteLine(e.Message);
            continue;
         }
         catch (DirectoryNotFoundException e) {
            Console.WriteLine(e.Message);
            continue;
         }
         catch (IOException e) {
            Console.WriteLine(e.Message);
            continue;
         }

         // Execute in parallel if there are enough files in the directory.
         // Otherwise, execute sequentially.Files are opened and processed
         // synchronously but this could be modified to perform async I/O.
         try {
            if (files.Length < procCount) {
               foreach (var file in files) {
                  action(file);
                  fileCount++;                            
               }
            }
            else {
               Parallel.ForEach(files, () => 0, (file, loopState, localCount) =>
                                            { action(file);
                                              return (int) ++localCount;
                                            },
                                (c) => {
                                          Interlocked.Add(ref fileCount, c);                          
                                });
            }
         }
         catch (AggregateException ae) {
            ae.Handle((ex) => {
                         if (ex is UnauthorizedAccessException) {
                            // Here we just output a message and go on.
                            Console.WriteLine(ex.Message);
                            return true;
                         }
                         // Handle other exceptions here if necessary...

                         return false;
            });
         }

         // Push the subdirectories onto the stack for traversal.
         // This could also be done before handing the files.
         foreach (string str in subDirs)
            dirs.Push(str);
      }

      // For diagnostic purposes.
      Console.WriteLine("Processed {0} files in {1} milleseconds", fileCount, sw.ElapsedMilliseconds);
   }

Answer 3

Unfortunately there's no hidden managed or Win32 API that would allow you to get the size of a folder on disk without recursing through it, otherwise Windows Explorer would definitely have taken advantage of it.

Here's a sample method that would parallelize the work which you could compare against a standard non-parallel recursive function to achieve the same:

private static long GetFolderSize(string sourceDir)
{
    long size = 0;
    string[] fileEntries = Directory.GetFiles(sourceDir);

    foreach (string fileName in fileEntries)
    {
        Interlocked.Add(ref size, (new FileInfo(fileName)).Length);
    }

    var subFolders = Directory.EnumerateDirectories(sourceDir);

    var tasks = subFolders.Select(folder => Task.Factory.StartNew(() =>
    {
        if ((File.GetAttributes(folder) & FileAttributes.ReparsePoint) != FileAttributes.ReparsePoint)
        {
            Interlocked.Add(ref size, (GetFolderSize(folder)));
            return size;
        }
        return 0;
    }));

    Task.WaitAll(tasks.ToArray());

    return size;
}

This example will not consume lots of memory unless you have millions of files inside a single folder.

Answer 4

Using the Microsoft Scripting Runtime seems to be about 90% faster:

var fso = new Scripting.FileSystemObject();
double size = fso.GetFolder(path).Size;

Reference: What is the fastest way to calculate a Windows folders size?