Walking through directory while controling depth - C#

Question

I need to be able to get all files from a directory and sub directories, but I would like to give the user the option to choose the depth of sub-directories. I.e., not just current directory or all directories, but he should be able to choose a depth of 1,2,3,4 directories etc.

I've seen many examples of walking through directory trees and none of them seemed to address this issue. And personally, I get confused with recursion... (which I currently use). I am not sure how I would track the depth during a recursive function.

Any help would be greatly appreciated.

Thanks, David

Here is my current code (which I found here):

    static void FullDirList(DirectoryInfo dir, string searchPattern, string excludeFolders, int maxSz, string depth)
    {

        try
        {
            foreach (FileInfo file in dir.GetFiles(searchPattern))
            {

                if (excludeFolders != "")
                    if (Regex.IsMatch(file.FullName, excludeFolders, RegexOptions.IgnoreCase)) continue;

                myStream.WriteLine(file.FullName);
                MasterFileCounter += 1;

                if (maxSz > 0 && myStream.BaseStream.Length >= maxSz)
                {
                    myStream.Close();
                    myStream = new StreamWriter(nextOutPutFile());
                }

            }
        }
        catch
        {
            // make this a spearate streamwriter to accept files that failed to be read.
            Console.WriteLine("Directory {0}  \n could not be accessed!!!!", dir.FullName);
            return;  // We alredy got an error trying to access dir so dont try to access it again
        }

        MasterFolderCounter += 1;

        foreach (DirectoryInfo d in dir.GetDirectories())
        {
            //folders.Add(d);
            // if (MasterFolderCounter > maxFolders) 
            FullDirList(d, searchPattern, excludeFolders, maxSz, depth);
        }

    }

Answer 1

using a maxdepth varibale that could be decremented each recursive call and then you cannot just return once reached the desired depth.

static void FullDirList(DirectoryInfo dir, string searchPattern, string excludeFolders, int maxSz, int maxDepth)
{

    if(maxDepth == 0)
    {
        return;
    }

    try
    {
        foreach (FileInfo file in dir.GetFiles(searchPattern))
        {

            if (excludeFolders != "")
                if (Regex.IsMatch(file.FullName, excludeFolders, RegexOptions.IgnoreCase)) continue;

            myStream.WriteLine(file.FullName);
            MasterFileCounter += 1;

            if (maxSz > 0 && myStream.BaseStream.Length >= maxSz)
            {
                myStream.Close();
                myStream = new StreamWriter(nextOutPutFile());
            }

        }
    }
    catch
    {
        // make this a spearate streamwriter to accept files that failed to be read.
        Console.WriteLine("Directory {0}  \n could not be accessed!!!!", dir.FullName);
        return;  // We alredy got an error trying to access dir so dont try to access it again
    }

    MasterFolderCounter += 1;

    foreach (DirectoryInfo d in dir.GetDirectories())
    {
        //folders.Add(d);
        // if (MasterFolderCounter > maxFolders) 
        FullDirList(d, searchPattern, excludeFolders, maxSz, depth - 1);
    }

}

Answer 2

Let's start out by refactoring the code a little bit to make its work a little easier to understand.

So, the key exercise here is to recursively return all of the files that match the patterns required, but only to a certain depth. Let's get those files first.

public static IEnumerable<FileInfo> GetFullDirList(
    DirectoryInfo dir, string searchPattern, int depth)
{
    foreach (FileInfo file in dir.GetFiles(searchPattern))
    {
        yield return file;
    }

    if (depth > 0)
    {
        foreach (DirectoryInfo d in dir.GetDirectories())
        {
            foreach (FileInfo f in GetFullDirList(d, searchPattern, depth - 1))
            {
                yield return f;
            }
        }
    }
}

This is just simplified the job of recursing for your files.

But you'll notice that it didn't exclude files based on the excludeFolders parameter. Let's tackle that now. Let's start building FullDirList.

The first line would be

    var results =
        from fi in GetFullDirList(dir, searchPattern, depth)
        where String.IsNullOrEmpty(excludeFolders)
            || !Regex.IsMatch(fi.FullName, excludeFolders, RegexOptions.IgnoreCase)
        group fi.FullName by fi.Directory.FullName;

This goes and gets all of the files, restricts them against excludeFolders and then groups all the files by the folders they belong to. We do this so that we can do this next:

    var directoriesFound = results.Count();
    var filesFound = results.SelectMany(fi => fi).Count();

Now I noticed that you were counting MasterFileCounter & MasterFolderCounter.

You could easily write:

    MasterFolderCounter+= results.Count();
    MasterFileCounter += results.SelectMany(fi => fi).Count();

Now, to write out these files it appears you are trying to aggregate the file names into separate files, but keeping a maximum length (maxSz) of the file.

Here's how to do that:

    var aggregateByLength =
        results
            .SelectMany(fi => fi)
            .Aggregate(new [] { new StringBuilder() }.ToList(),
                (sbs, s) =>
                {
                    var nl = s + Environment.NewLine;
                    if (sbs.Last().Length + nl.Length > maxSz)
                    {
                        sbs.Add(new StringBuilder(nl));
                    }
                    else
                    {
                        sbs.Last().Append(nl);
                    }
                    return sbs;
                });

Writing the files now becomes extremely simple:

    foreach (var sb in aggregateByLength)
    {
        File.WriteAllText(nextOutPutFile(), sb.ToString());
    }

So, the full thing becomes:

static void FullDirList(
    DirectoryInfo dir, string searchPattern, string excludeFolders, int maxSz, int depth)
{
    var results =
        from fi in GetFullDirList(dir, searchPattern, depth)
        where String.IsNullOrEmpty(excludeFolders)
            || !Regex.IsMatch(fi.FullName, excludeFolders, RegexOptions.IgnoreCase)
        group fi.FullName by fi.Directory.FullName;

    var directoriesFound = results.Count();
    var filesFound = results.SelectMany(fi => fi).Count();

    var aggregateByLength =
        results
            .SelectMany(fi => fi)
            .Aggregate(new [] { new StringBuilder() }.ToList(),
                (sbs, s) =>
                {
                    var nl = s + Environment.NewLine;
                    if (sbs.Last().Length + nl.Length > maxSz)
                    {
                        sbs.Add(new StringBuilder(nl));
                    }
                    else
                    {
                        sbs.Last().Append(nl);
                    }
                    return sbs;
                });

    foreach (var sb in aggregateByLength)
    {
        File.WriteAllText(nextOutPutFile(), sb.ToString());
    }
}