How to detect Loop while Enumerating Reparse Points/Junctions in C#?

Question

I am using Directory.EnumerateFiles() to Enumerate files in my program. I am having issue while enumerating the "Application Data" folder (available at : C:\Program Data\Application Data).

The EnumerateFiles() goes into loop as "Application Data" folder has one more shortcut with name "Application Data" in it and it has one more inside it and so on.

After few iterations, path becomes long enough and EnumerateFiles() comes out with PathTooLongException.

How do I avoid this loop and enumerate remaining files?

Code:

    private void EnumerateRecursively(DirectoryInfo dirInfo, string searchPattern, EnumerationData data)
    {
        try
        {
            EnumerateDirectory(dirInfo, searchPattern, data, SearchOption.TopDirectoryOnly);

            foreach (DirectoryInfo dir in dirInfo.EnumerateDirectories())
                {
                    EnumerateDirectory(dir, searchPattern, data, SearchOption.AllDirectories);
                }
        }
        catch (PathTooLongException ex)
        {
            WriteTrace(FormatExceptionData(ex));
        }
        catch (Exception ex)
        {
            WriteTrace(FormatExceptionData(ex));
        }
    }

private void EnumerateDirectory(DirectoryInfo dir, string searchPattern, EnumerationData data, SearchOption searchOption)
    {
        try
        {
            foreach (FileInfo fileInfo in dir.EnumerateFiles(searchPattern, searchOption))
            {
                //Do something
            }
        }
        catch (PathTooLongException ex)
        {
            WriteTrace(FormatExceptionData(ex));
        }
        catch(UnauthorizedAccessException ex)
        {
            WriteTrace(FormatExceptionData(ex));
        }
        catch (Exception ex)
        {
            WriteTrace(FormatExceptionData(ex));
        }
    }

EDIT: As I found out, what I refer here as loop is actually called Reparse Points/Junctions. Sometimes these points' target have copy of this reparse point which leads to endless loop during enumeration.

@JanusPienaar: I can not ignore all Applicaton Data folders as User can create his own Application Data folder which should be enumerated properly. — Himanshu1983, Jul 20 '17 at 11:46
What is the **exact** path when it throws `PathTooLongException`? — mjwills, Jul 20 '17 at 11:54
[Find out](https://stackoverflow.com/questions/1485155/check-if-a-file-is-real-or-a-symbolic-link) if a directory is actually a symbolic link or junction; if so, stop enumerating its children. Or, more elaborate, check if you already saw the "target path" of it, and stop if you did. — Christian.K, Jul 20 '17 at 12:02
@mjwills: It would be something like "C:\ProgramData\Application Data\Application Data\Application Data\Application Data\Application Data\..." until it reaches the maximum path length allowed under windows. — Himanshu1983, Jul 20 '17 at 12:07
Is it **something like** that or **exactly** that? The exact value will really help us to help you. — mjwills, Jul 20 '17 at 12:07
@Christian.K: Can be done but this will also avoid other links which do not have loop? I cannot skip them. — Himanshu1983, Jul 20 '17 at 12:10
Hence the 2nd part: only stop if you have already seen a path; that is keep track of all of them in a list or such. That way you wouldn't stop on non-loops. — Christian.K, Jul 20 '17 at 12:11
@mjwills: This case can occur whenever there is a loop pointing to itself. This is **exact** path for one such case. — Himanshu1983, Jul 20 '17 at 12:12
Does it work if you change `EnumerateDirectories` to `GetDirectories` and `EnumerateFiles` to `GetFiles`? — mjwills, Jul 20 '17 at 12:17
@mjwills: AFAIK, the only difference between `EnumerateFiles()` and `GetFiles()` is lazy evaluation of the later. Speed of evaluation is important for us as we want to start doing other processing as soon as enumeration starts giving out files, so using `GetFiles()` is ruled out. — Himanshu1983, Jul 20 '17 at 12:25
So you tried it and it didn't work? If you can confirm whether it does or doesn't work then it may give us some more information. https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/file-system/how-to-iterate-through-a-directory-tree implies it _may_ work. — mjwills, Jul 20 '17 at 12:26
@mjwills: I was using it in pre .NET 4.0 era but now since `EnumerateFiles()` is faster, we switched to it. As I said, it is important for our product to reduce enumeration time, so cannot get back to `GetFiles()`as we have to wait till it returns the array after complete enumeration. — Himanshu1983, Jul 20 '17 at 12:32
I am not suggesting you leave it this way forever. But **for now** can you try my suggestion? This process gets really slow if every time we ask you to try something you resist. :) — mjwills, Jul 20 '17 at 12:32
@mjwills: Let me go through the link shared by you. I will come back with my findings. Thanks for help. :) — Himanshu1983, Jul 20 '17 at 12:35

score 0 · Answer 1 · answered Jul 20 '17 at 11:44

0

Enumerate files has an overload where you can tell it not to search sub-directories (the third, searchOption parameter):

https://msdn.microsoft.com/en-us/library/dd383571(v=vs.110).aspx

answered Jul 20 '17 at 11:44

Adam Diament

4,290
3
34
55

1

IIRC, it is the *default behaviour* to not search sub-directories - i.e. `SearchOption.TopDirectoryOnly` is the effective default – Marc Gravell Jul 20 '17 at 11:45
Problem is not search criteria as such but where to stop enumerating as there is a loop. I will still encounter this problem even if I go through the directory one by one using `SearchOption.TopDirectoryOnly`. Also, the solution has to work for other normal cases where user can create some valid hierarchy with same folder names like C:\abc\abc\abc. – Himanshu1983 Jul 20 '17 at 11:52
please edit your question to add code to include exactly how you are using the method, thanks – Adam Diament Jul 20 '17 at 11:55
The problem is you are specifically telling it to use AllDirectories. Change the third parameter to SearchOption.TopDirectoryOnly – Adam Diament Jul 20 '17 at 12:06
Or if you need to do that, your next best bet may be to add a "if dir.Name != "Application Data" if statement – Adam Diament Jul 20 '17 at 12:07
@AdamDiament: It is possible to create normal `Application Data` folder so using this logic other folders with the same name also will get filtered. – Himanshu1983 Jul 20 '17 at 12:28
Ok in this case you need to define a condition for when an ApplicationData folder is valid and when it is not, and do an if statement to filter for it. For example "If it is called application data but it doesn't have any files inside" it is not valid. If it was me I would take a step back from my code, define the use cases that are valid and invalid, and create a flow diagram of what should happen. Then write my code accordingly. – Adam Diament Jul 20 '17 at 12:36

score -2 · Answer 2 · edited Jul 20 '17 at 16:30

I am able to get the data with out any issue even with short cuts(dir1 shortcut in dir1 / dir shortcut in it sub folder) in its folder/sub folders here is the code.

static void Main(string[] args)
{
    string sourceDirectory = @"C:\dir1";
    string archiveDirectory = @"C:\dir2";

    try
    {
        var txtFiles = Directory.EnumerateFiles(sourceDirectory, "*.txt", SearchOption.AllDirectories);

        foreach (string currentFile in txtFiles)
        {
            string fileName = currentFile.Split("\\".ToCharArray()).Last();
            Directory.Move(currentFile, Path.Combine(archiveDirectory, fileName));
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e.Message);
    }
}

How to detect Loop while Enumerating Reparse Points/Junctions in C#?

2 Answers2