6

I am trying to find out that information. Apparently Microsoft has not provided code to read processes stdout and stderr reliably without deadlocks, exceptions and other issues.

Deadlock problem

http://www.codeducky.org/process-handling-net/

A less-obvious problem is that of deadlocking. All three process streams (in, out, and error) are finite in how much content they can buffer. If the internal buffer fills up, then whoever is writing to the stream will block. In this code, for example, we don’t read from the out and error streams until after the process has exited. That means that we could find ourselves in a case where external process exhausts it’s error buffer. In that case, external process would block on writing to standard error, while our .NET app is blocked reading to the end of standard out. Thus, we’ve found ourselves in a deadlock!

There is no reliable code on the internet

There is a 265 times upvoted answer, but I am not going to use it because it suffers from ObjectDisposeException and requires timeouts:
ProcessStartInfo hanging on "WaitForExit"? Why?

It was probably upvoted before it was discovered it can lead to ObjectDisposeException.

What are chances of experiencing deadlock?

I want to know what are the chances of getting into deadlock situation. For that I neeed to know the buffer sizes for stderr and stdout under windows 7.

How to find them out? I thought I could cat several files to see what approx file size will cause problems, but there is no cat under windows 7. I even tried using git cat-file but it has bad documentation with no usage examples and no one has answered a question regarding that: https://stackoverflow.com/questions/43203902/how-to-git-cat-file-a-file-in-current-dir

Community
  • 1
  • 1
Marko Avlijaš
  • 1,579
  • 13
  • 27
  • 1
    [It looks like the buffer size might be 4 kilobytes](https://github.com/Microsoft/referencesource/blob/master/System/services/monitoring/system/diagnosticts/Process.cs) - see lines 2165 and 2169 – ProgrammingLlama Apr 04 '17 at 10:37
  • The first article you mentioned explains how to avoid the deadlock (read from both streams asynchronously). Also, there is [type](https://ss64.com/nt/type.html) command which is roughly similar to cat. – Evk Apr 04 '17 at 10:37
  • @Evk - that was done in the answer I linked to, but it leads to `ObjectDisposeException` and requires timeout which is not handy when writing git wrapper code (think pull and push which access the internet). – Marko Avlijaš Apr 04 '17 at 10:40
  • 2
    These buffers exist in the process you started, not in .NET and not in the Process class. Actual size entirely depends on the toolchain used to build the program, 4096 bytes is a very typical value. But the programmer could have changed it, setvbuf() is the boilerplate C library function used to do so. If you want to point an accusing finger then you'll have to aim it at Ken Thompson, I/O redirection is a horribly broken feature that should have stayed in Unix. – Hans Passant Apr 04 '17 at 10:44
  • I think the buffers are in the target process rather than the launching one, and the sizes would depend on what runtime (e.g. C runtime) – Damien_The_Unbeliever Apr 04 '17 at 10:44
  • 1
    Code in the answer is indeed broken, but it is fixable and there is a fix in another answer a bit down. Also, it's not the same code which is mentioned in the first link article. I'd suggest you to try code from first article which seems reasonable, or even try the library (MediallionShell) they suggest there. – Evk Apr 04 '17 at 10:48
  • You can always get it once process has been started using long len = process.StandardOutput.BaseStream.Length; – jdweng Apr 04 '17 at 10:52
  • @Evk - which answer bellow fixes the accepted answer? – Marko Avlijaš Apr 04 '17 at 10:55
  • @jdweng - I get `System.NotSupportedException` with `Additional information: Stream does not support seeking.` – Marko Avlijaš Apr 04 '17 at 10:56
  • Several answers, for example one by Karol Tyl. – Evk Apr 04 '17 at 10:58
  • And is easily findable by searching on the page for `ObjectDisposed` - it's one of only two instances of that string of characters. Don't expect to have everything spoon-fed to you. – Damien_The_Unbeliever Apr 04 '17 at 11:00

2 Answers2

5

John is correct in his comment, it's hardcoded to 4096 bytes.

https://github.com/Microsoft/referencesource/blob/master/System/services/monitoring/system/diagnosticts/Process.cs

    if (startInfo.RedirectStandardInput) {
            standardInput = new StreamWriter(new FileStream(standardInputWritePipeHandle, FileAccess.Write, 4096, false), Console.InputEncoding, 4096);
            standardInput.AutoFlush = true;
        }
        if (startInfo.RedirectStandardOutput) {
            Encoding enc = (startInfo.StandardOutputEncoding != null) ? startInfo.StandardOutputEncoding : Console.OutputEncoding;
            standardOutput = new StreamReader(new FileStream(standardOutputReadPipeHandle, FileAccess.Read, 4096, false), enc, true, 4096);
        }
        if (startInfo.RedirectStandardError) {
            Encoding enc = (startInfo.StandardErrorEncoding != null) ? startInfo.StandardErrorEncoding : Console.OutputEncoding;
            standardError = new StreamReader(new FileStream(standardErrorReadPipeHandle, FileAccess.Read, 4096, false), enc, true, 4096);

}

I have tested this by running cmd.exe type path\to\file.html command, but it printed reduced output, so I wrote a one line ruby script to open a file and print it to stdout. It failed with 4373 bytes file and worked with 3007 bytes file. It just hangs, deadlocks.

Really pathetic process handling from Microsoft IMO.

Marko Avlijaš
  • 1,579
  • 13
  • 27
0

Update for .NET Core and later: There is no longer a buffer limit.

This is the class that handles the reading nowadays: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/AsyncStreamReader.cs. It reads in up to 1K at a time, but immediately adds it to growable heap memory structures.

Matt Tsōnto
  • 1,518
  • 1
  • 15
  • 30