Recurse a folder in order using IEnumerable

Question

I've written a function to recurse a folder structure, but it's not working as intended. The intended behavior is to return the files of the root folder passed, then go into the first folder, return those files, then go into the first subfolder of the first folder and list those files, etc, etc.

Instead, I've managed to get it to return all the Level 1 folders first, then all the Level 2 folders then all the Level 3 etc etc. I can see what is wrong but I'm not sure how to resolve it.

How can I rewrite this to have my expected course of action?

static IEnumerable<string> recurseFolder(String Folder)
{
    if (Directory.Exists(Folder))
    {
        string[] files = null;
        string[] dirs = null;
        try { files = Directory.GetFiles(Folder); } catch (Exception) { }

        if (files != null)
        {
            foreach (var item in files)
            {
                yield return item;
            }
        }

        try { dirs = Directory.GetDirectories(Folder); } catch (Exception) { }

        if (dirs != null)
        {
            foreach (var dir in dirs)
            {
                foreach (var item in recurseFolder(dir))
                {
                    yield return item;
                }

            }
        }
    }

}

Update 1 This may be an inconsistency with dotnet core on Linux and Windows, this is my results on Windows and what I was expecting it would look like.

d:\root\Folder A\File 1.txt
d:\root\Folder A\Folder B\File 2.txt
d:\root\Folder A\Folder B\Folder C\File 3.txt
d:\root\Folder A\Folder B\Folder D\File 4.txt
d:\root\Folder A\Folder E\File 5.txt
d:\root\Folder A\Folder F\File 6.txt
d:\root\Folder A\Folder F\Folder G\File 7.txt
d:\root\Folder H\File 8.txt
d:\root\Folder H\Folder I\File 9.txt
d:\root\Folder H\Folder I\Folder J\File 10.txt
d:\root\Folder H\Folder I\Folder K\File 11.txt
d:\root\Folder H\Folder L\File 12.txt
d:\root\Folder M\File 13.txt

This is my results on Linux

/home/x/Root/Folder A/File 1.txt
/home/x/Root/Folder A/Folder E/File 5.txt
/home/x/Root/Folder A/Folder B/File 2.txt
/home/x/Root/Folder A/Folder B/Folder D/File 4.txt
/home/x/Root/Folder A/Folder B/Folder C/File 3.txt
/home/x/Root/Folder A/Folder F/File 6.txt
/home/x/Root/Folder A/Folder F/Folder G/File 7.txt
/home/x/Root/Folder H/File 8.txt
/home/x/Root/Folder H/Folder I/File 9.txt
/home/x/Root/Folder H/Folder I/Folder K/File 11.txt
/home/x/Root/Folder H/Folder I/Folder J/File 10.txt
/home/x/Root/Folder H/Folder L/File 12.txt
/home/x/Root/Folder M/File 13.txt

Maybe my code is working but I just need to manually sort the directories into alphabetical order, I wonder if linux is returning them in created or modified date order.

You might want to know that `Directory.GetFiles` has an overload that takes a `SearchOption` and you can tell it to search all sub directories. https://msdn.microsoft.com/en-us/library/ms143316(v=vs.110).aspx — juharr, Mar 05 '18 at 14:20
Why don't you pass SearchOption.AllDirectoreis to recurse directories in GetFiles? Are you trying to avoid folders to which you have no access, perhaps trying to list all files in `C:` or inside `C:\Windows`? — Panagiotis Kanavos, Mar 05 '18 at 14:21
Avoid catching Exception - would you really want to catch OutOfMemoryException for example? — alastairtree, Mar 05 '18 at 14:23
Have a static `List<>` and then get all file names `Directory.GetFiles` and get all folder names `Directory.GetDirectories` and then for each folder call the same method again. Keep inserting file names in your static list — FortyTwo, Mar 05 '18 at 14:23
Could you include a example of the directory structure you are working with and the results you get and the results you want? Right now it looks to me like it should give you the results you want. — juharr, Mar 05 '18 at 14:23
@juharr I don't think it guarantees returning the files in ascending with folder tree level. — , Mar 05 '18 at 14:26
Very related to https://stackoverflow.com/questions/929276/how-to-recursively-list-all-the-files-in-a-directory-in-c — alastairtree, Mar 05 '18 at 14:27
@alastairtree Nope. He also wants recurse all files but they must be ordered with folder levels. — , Mar 05 '18 at 14:29
@donggas90 Sometimes it's easier to explain with examples vs having misunderstanding with descriptions. — juharr, Mar 05 '18 at 14:29
In my opinion, it cannot be done with `yield`. But it is easy with cache all files, sorting and much more cost. — , Mar 05 '18 at 14:34
Sorry it's been a busy day at work and I didn't expect all these responses so quickly! I'm not using SearchOption.AllDirectories because of the permissions issue, I'm dealing with quite large folder structures and doing a hash on all files so I don't want to build it first and then return, hence yield. I will try and show some examples of what it is currently doing vs how I want it to work. — Matt, Mar 05 '18 at 17:36
Also apologies for not being specific about wanting to use Yield to reduce the time to process the first item. — Matt, Mar 05 '18 at 17:39

InteXX · Answer 1 · 2018-03-05T16:30:00.087

I haven't used this for a good while, but if memory serves I believe it may give you the results you seek. It uses flat recursion.

This is a rough conversion from VB by JustDecompile, so hopefully the syntax is correct.

Recurser.cs

using System;
using System.Collections.Generic;
using System.IO;
using System.Runtime.CompilerServices;

namespace Recurser
{
    public class Recurser
    {
        public Files Files
        {
            get;
        }

        public Folders Folders
        {
            get;
        }

        public string RootFolder
        {
            get;
        }

        public string SearchPattern
        {
            get;
        }

        public Recurser(string RootFolder) : this(RootFolder, "*.*")
        {
        }

        public Recurser(string RootFolder, string SearchPattern)
        {
            this.Folders = new Folders();
            this.Files = new Files();
            this.SearchPattern = SearchPattern;
            this.RootFolder = RootFolder;
        }

        public void Start()
        {
            this.Folders.Clear();
            this.Files.Clear();
            int iCount = 0;
            if (Directory.Exists(this.RootFolder))
            {
                this.Folders.Add(new DirectoryInfo(this.RootFolder));
                while (iCount < this.Folders.Count)
                {
                    this.Folders.AddRange(this.Folders[iCount].GetDirectories());
                    this.Files.AddRange(this.Folders[iCount].GetFiles());
                    iCount = checked(iCount + 1);
                }
            }
        }
    }
}

Folders.cs

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Text;

namespace Recurser
{
    public class Folders : List<DirectoryInfo>
    {
        public Folders()
        {
        }

        public override string ToString()
        {
            base.Sort(new FolderComparer());
            StringBuilder stringBuilder = new StringBuilder();
            base.ForEach((DirectoryInfo Folder) => stringBuilder.AppendLine(Folder.FullName));
            return stringBuilder.ToString();
        }

        public IEnumerable<string> ToStrings()
        {
            base.Sort(new FolderComparer());
            return this.Select((DirectoryInfo Folder) => Folder.fullName);
        }
    }
}

Files.cs

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Text;

namespace Recurser
{
    public class Files : List<FileInfo>
    {
        public Files()
        {
        }

        public override string ToString()
        {
            base.Sort(new FileComparer());
            StringBuilder stringBuilder = new StringBuilder();
            base.ForEach((FileInfo File) => stringBuilder.AppendLine(File.FullName));
            return stringBuilder.ToString();
        }

        public IEnumerable<string> ToStrings()
        {
            base.Sort(new FileComparer());
            return this.Select((FileInfo File) => File.fullName);
        }
    }
}

FolderComparer.cs

using System;
using System.Collections.Generic;
using System.IO;

namespace Recurser
{
    public class FolderComparer : IComparer<DirectoryInfo>
    {
        public FolderComparer()
        {
        }

        public int Compare(DirectoryInfo X, DirectoryInfo Y)
        {
            return X.FullName.CompareTo(Y.FullName);
        }
    }
}

FileComparer.cs

using System;
using System.Collections.Generic;
using System.IO;

namespace Recurser
{
    public class FileComparer : IComparer<FileInfo>
    {
        public FileComparer()
        {
        }

        public int Compare(FileInfo X, FileInfo Y)
        {
            return X.FullName.CompareTo(Y.FullName);
        }
    }
}

Wow, that's very comprehensive, maybe I should have been more specific however, I want to use Yield to return the results on the fly as it's a very large folder/file structure and I want to make it start straight away and not spend a long time building the structure before hand. — Matt, Mar 05 '18 at 17:39
@Matt — It appears you're correct; Linux must be returning them in some order other than `FullName`. You could run a small test to confirm this. Assuming so, your only defense against it will be to sort the entire list after it's built—which of course means dispensing with `yield`. So something's gotta give: either your sorting requirement or `yield`, I'm afraid. If thread blocking is a serious issue, you could use `async/await`. — InteXX, Mar 06 '18 at 03:07

score 0 · Accepted Answer · answered Mar 12 '18 at 15:49

Turned out since my understanding of the issue was incorrect the fix was quite easy, I just added in some OrderBy statements on the files and the dir arrays.

static IEnumerable<string> recurseFolder(String Folder)
{
    if (Directory.Exists(Folder))
    {
        string[] files = null;
        string[] dirs = null;
        try { files = Directory.GetFiles(Folder); } catch (Exception) { }

        if (files != null)
        {
            foreach (var item in files.OrderBy(o => o))
            {
                yield return item;
            }
        }

        try { dirs = Directory.GetDirectories(Folder); } catch (Exception) { }

        if (dirs != null)
        {
            foreach (var dir in dirs.OrderBy(o =>o))
            {
                foreach (var item in recurseFolder(dir))
                {
                    yield return item;
                }
            }
        }
    }
}

Recurse a folder in order using IEnumerable

2 Answers2