C# - How to proccess data from Process.StandardOutput when the data is around 400 MB

Question

I am trying to retrieve list of files from a server with the windows command - "DIR /S/B" The output is huge (around 400 MB). Now when I tried retrieve it with below approach, its taking hours to process. Is there any faster way to do it.

string path = args[0];
var start = DateTime.Now;

System.Diagnostics.ProcessStartInfo procStartInfo =
    new System.Diagnostics.ProcessStartInfo("cmd", "/c " + "dir /s/b " + path );
procStartInfo.RedirectStandardOutput = true;
procStartInfo.UseShellExecute = false;
procStartInfo.CreateNoWindow = true;
System.Diagnostics.Process proc = new System.Diagnostics.Process();
proc.StartInfo = procStartInfo;
proc.Start();

//string [] result = proc.StandardOutput.ReadToEnd().Split('\n'); ;

StreamWriter writer = new StreamWriter("FileList.lst");
while (proc.StandardOutput.EndOfStream != true)
{
    writer.WriteLine(proc.StandardOutput.ReadLine());
    writer.Flush();
}
writer.Close();

Justin Niessner · Accepted Answer · 2011-09-27T14:16:36.057

7

Why not use DirectoryInfo.GetFiles?

I'm guessing quite a bit of your time now is being eaten up by the command executing, not the .NET code. It'll take dir a long time to write that much data to a stream in sequence. You then use String.Split which is also going to choke on that much data.

By using DirectoryInfo.GetFiles, you should be able to get all the file names in a single line (and you could also get other information about the files this way):

var files = (new DirectoryInfo(path)
                .GetFiles("*.*", SearchOption.AllDirectories)
                .Select(fi => fi.Name);

If you're really only concerned about filenames, you could use:

var fileNames = Directory.GetFiles(path, "*.*", SearchOption.AllDirectories);

edited Sep 27 '11 at 14:16

answered Sep 27 '11 at 14:10

Justin Niessner

242,243
40
408
536

I tried the DirectoryInfo approach, but its getting failed for some of the folder in windows directory. It gives an exception.. Access Denied to the file/folder. While DIR runs smooth for the same Files/Folders. – Mayur J Sep 27 '11 at 14:15
@MayurJ I have had this issue before, here is the question I posted on it, maybe the answers will help you for this http://stackoverflow.com/questions/6762101/c-file-permission-exclusion – Bali C Sep 27 '11 at 14:36
In .Net 4.0 or higher, recommend using EnumerateFiles rather than GetFiles as GetFiles will not return until the entire array has been filled, whereas with EnumerateFiles you can begin iterating of the the files immediately. – Chris Dunaway Sep 27 '11 at 15:24
@Chris - Unfortunetly I am using .NET 3.5 – Mayur J Sep 28 '11 at 07:05

score 1 · Answer 2 · answered Sep 27 '11 at 14:11

1

You're reinventing the wheel. Add a reference to System.IO and use the DirectoryInfo and FileInfo classes.

answered Sep 27 '11 at 14:11

The Evil Greebo

7,013
3
28
55

score 0 · Answer 3 · answered Sep 27 '11 at 14:12

0

When you say recieve, do you simply mean list the files in the directory?

If so, can you not use the Directory.GetFiles() method?

        // Only get files that begin with the letter "c."
        string[] dirs = Directory.GetFiles(@"c:\", "c*");
        Console.WriteLine("The number of files starting with c is {0}.", dirs.Length);
        foreach (string dir in dirs) 
        {
            Console.WriteLine(dir);
        }

From MSDN

answered Sep 27 '11 at 14:12

dougajmcdonald

19,231
12
56
89

I wanted to list all the files in the directory and sub-directory, without getting the access denied exception for the syatem folder. Just like the windows DIR command. (Try giving c:\windows\system32 as the directory and you will get the access denied error) – Mayur J Sep 27 '11 at 14:32

C# - How to proccess data from Process.StandardOutput when the data is around 400 MB

3 Answers3