Why does StandardOutput.Read() block when StartInfo.RedirectStandardInput is set to true?

Question

I'm having a hard time deciphering the MSDN doc about Process.StandardOutpout as to if the Read(Char[], Int32, Int32) method blocks or not. My understanding is that it shouldn't block, but it seems like it does when I set RedirectStandardInput to true.

Does anybody have experience with this; or some explanation for the issue I'm having?

The context here is that I don't want to wait for a full line (ie with a line terminator), or for the process to exit before reading standard output. Also I don't want to use callbacks. I want to read StdOut synchronously as the process writes to it.

Here is a simplified version of my code:

string command = @"C:\flex_sdks\flex_sdk_4.5.1.21328\bin\fcsh.exe";
Process p = new Process();
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardInput = false; # <-- if I set this to true, then 
                                           # the program hangs on
                                           # p.StandardOutput.Read later on
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.CreateNoWindow = true;
p.StartInfo.FileName = command;
p.Start();

StringBuilder sb_stdout = new StringBuilder(1024);
char[] buffer = new char[64];
int nb_bytes_read;
while (true) {
    do {
        nb_bytes_read = p.StandardOutput.Read(buffer, 0, buffer.Length);
        sb_stdout.Append(new string(buffer, 0, nb_bytes_read));
    } while (nb_bytes_read > 0);
    if (sb_stdout.ToString().EndsWith("\n(fcsh) "))
        break;
    Thread.Sleep(20);
}

Update

Based on my (probably bad) assumption that Process.StandardOutput is broken when used:

• with stdin redirected; and,
• while reading something else than terminated lines from stdout or stderr,

I decided to try using Windows' API directly. I added an answer with such code; it works fine (at least for now).

Another update

I created a blog entry with the code I now use.

Answer 1

i fought and fought with this just last week actually... for some reason anything other than the Read() call (ReadToEnd() was not what i needed) seemed to block and never return. here's what i did to finally get it to "work":

snip 1:

    private bool ThreadExited = true;
    private bool ExitThread = false;
    private void ReadThread()
    {
        while (!ExitThread)
        {
            string Output = "";

            int CharacterInt = myProcess.StandardOutput.Read();

            while (CharacterInt > 0)
            {
                char Character = (char)CharacterInt;
                Output += Character;

                var MyDelegate = new delegateUpdateText(UpdateText);
                Invoke(MyDelegate, Output);

                Output = "";
                CharacterInt = myProcess.StandardOutput.Read();
            }

            System.Threading.Thread.Yield();
        }

        ThreadExited = true;
    }

snip 2:

    private void InitializeProcess()
    {
        ThreadExited = true;
        ExitThread = true;

        while (!ThreadExited)
            System.Threading.Thread.Sleep(1000);

        ThreadExited = false;
        ExitThread = false;

        myProcess = new Process();

        ProcessStartInfo PSI = myProcess.StartInfo;

        PSI.FileName = @"cmd.exe";
        PSI.UseShellExecute = false;
        PSI.RedirectStandardError = false;
        PSI.RedirectStandardInput = true;
        PSI.RedirectStandardOutput = true;
        PSI.CreateNoWindow = false;
        PSI.ErrorDialog = true;


        myProcess.StartInfo = PSI;

        myProcess.Exited += new EventHandler(myProcess_Exited);

        myProcess.EnableRaisingEvents = false;


        myProcess.Start();

        ReadThreadThread = new System.Threading.Thread(ReadThread);
        ReadThreadThread.Start();
    }
    private System.Threading.Thread ReadThreadThread;

that finally ended up working for me. in my case i was writing text to to a textbox but that should be easy enough to modify to something else. but anything else i did caused issues due to the blocks; for some reason even if i used reflection to get the number of bytes that were available, calling the ReadBlock() function would block. never did figure it out to my satisfaction.

Answer 2

After discussing this a bit with Ben Voigt, I decided to implement communicating with the process without using System.Diagnostics.Process. This is what I came up with for now and it works great, ie it works consistently every time, and nothing blocks or hangs.

I'm posting this as this might help anyone needing to read from stdout/stderr and to write to stdin of some created process while doing without System.Diagnostics.Process.

const UInt32 STARTF_USESTDHANDLES = 0x00000100;
const int HANDLE_FLAG_INHERIT = 1;

struct PROCESS_INFORMATION
{
    public IntPtr hProcess;
    public IntPtr hThread;
    public uint dwProcessId;
    public uint dwThreadId;
}

struct STARTUPINFO
{
    public uint cb;
    public string lpReserved;
    public string lpDesktop;
    public string lpTitle;
    public uint dwX;
    public uint dwY;
    public uint dwXSize;
    public uint dwYSize;
    public uint dwXCountChars;
    public uint dwYCountChars;
    public uint dwFillAttribute;
    public uint dwFlags;
    public short wShowWindow;
    public short cbReserved2;
    public IntPtr lpReserved2;
    public IntPtr hStdInput;
    public IntPtr hStdOutput;
    public IntPtr hStdError;
}

struct SECURITY_ATTRIBUTES
{
    public int length;
    public IntPtr lpSecurityDescriptor;
    [MarshalAs(UnmanagedType.Bool)]
    public bool bInheritHandle;
}

[DllImport("kernel32.dll")]
static extern bool CreateProcess(string lpApplicationName,
                                 string lpCommandLine,
                                 IntPtr lpProcessAttributes,
                                 IntPtr lpThreadAttributes,
                                 bool bInheritHandles,
                                 uint dwCreationFlags,
                                 IntPtr lpEnvironment,
                                 string lpCurrentDirectory,
                                 ref STARTUPINFO lpStartupInfo,
                                 out PROCESS_INFORMATION lpProcessInformation);

[DllImport("kernel32.dll", SetLastError = true)]
static extern bool CloseHandle(IntPtr hObject);

[DllImport("kernel32.dll", SetLastError = true)]
static extern bool CreatePipe(out IntPtr hReadPipe,
                              out IntPtr hWritePipe,
                              ref SECURITY_ATTRIBUTES lpPipeAttributes,
                              uint nSize);

[DllImport("kernel32", SetLastError = true)]
static extern unsafe bool ReadFile(IntPtr hFile,
                                   void* pBuffer,
                                   int NumberOfBytesToRead,
                                   int* pNumberOfBytesRead,
                                   IntPtr lpOverlapped);

[DllImport("kernel32.dll")]
static extern unsafe bool WriteFile(IntPtr hFile,
                                    void* pBuffer,
                                    int nNumberOfBytesToWrite,
                                    int* lpNumberOfBytesWritten,
                                    IntPtr lpOverlapped);

[DllImport("kernel32.dll")]
static extern bool SetHandleInformation(IntPtr hObject, int dwMask, uint dwFlags);

void OpenAndCloseFcsh()
{
    STARTUPINFO si = new STARTUPINFO();
    SECURITY_ATTRIBUTES sa = new SECURITY_ATTRIBUTES();
    PROCESS_INFORMATION pi = new PROCESS_INFORMATION();

    sa.bInheritHandle = true;
    sa.lpSecurityDescriptor = IntPtr.Zero;
    sa.length = Marshal.SizeOf(typeof(SECURITY_ATTRIBUTES));
    sa.lpSecurityDescriptor = IntPtr.Zero;

    IntPtr h_stdout_r, h_stdout_w;
    if (!CreatePipe(out h_stdout_r, out h_stdout_w, ref sa, 0))
        throw new Exception("bad");
    if (!SetHandleInformation(h_stdout_r, HANDLE_FLAG_INHERIT, 0))
        throw new Exception("bad");

    IntPtr h_stdin_r, h_stdin_w;
    if (!CreatePipe(out h_stdin_r, out h_stdin_w, ref sa, 0))
        throw new Exception("bad");
    if (!SetHandleInformation(h_stdin_w, HANDLE_FLAG_INHERIT, 0))
        throw new Exception("bad");

    si.wShowWindow = 0;
    si.cb = (uint)Marshal.SizeOf(si);
    si.dwFlags |= STARTF_USESTDHANDLES;
    si.hStdOutput = h_stdout_w;
    si.hStdError = h_stdout_w;
    si.hStdInput = h_stdin_r;

    string command = @"C:\flex_sdks\flex_sdk_4.5.1.21328_trimmed\bin\fcsh.exe";

    if (!CreateProcess(command, null, IntPtr.Zero, IntPtr.Zero, true, 0, IntPtr.Zero, null, ref si, out pi))
        throw new Exception("bad");

    Console.WriteLine("Process ID (PID): " + pi.dwProcessId);
    Console.WriteLine("Process Handle : " + pi.hProcess);

    // ****************************************************
    // let's interact with our process

    // first read to the prompt
    Console.WriteLine("read this from fcsh.exe:\r\n" + ReadTillPrompt(h_stdout_r));

    // write "help" to stdin
    byte[] bytes_to_write = Encoding.UTF8.GetBytes("help\r\n");
    Write(h_stdin_w, bytes_to_write, 0, bytes_to_write.Length);

    // then read to the prompt again
    Console.WriteLine("read this from fcsh.exe:\r\n" + ReadTillPrompt(h_stdout_r));

    // write "quit" to stdin
    bytes_to_write = Encoding.UTF8.GetBytes("quit\r\n");
    Write(h_stdin_w, bytes_to_write, 0, bytes_to_write.Length);

    // ****************************************************

    if (!CloseHandle(pi.hProcess))
        throw new Exception("bad");
    if (!CloseHandle(pi.hThread))
        throw new Exception("bad");
    if (!CloseHandle(h_stdout_w))
        throw new Exception("bad");
    if (!CloseHandle(h_stdin_w))
        throw new Exception("bad");
}

public string ReadTillPrompt(IntPtr h_stdout_r)
{
    StringBuilder sb = new StringBuilder(1024);
    byte[] buffer = new byte[128];

    int nb_bytes_read;
    while (true) {
        nb_bytes_read = Read(h_stdout_r, buffer, 0, buffer.Length);
        sb.Append(Encoding.UTF8.GetString(buffer, 0, nb_bytes_read));
        if (sb.ToString().EndsWith("\n(fcsh) "))
            break;
        Thread.Sleep(20);
    }
    return sb.ToString();
}

public unsafe int Read(IntPtr h, byte[] buffer, int index, int count)
{
    int n = 0;
    fixed (byte* p = buffer) {
        if (!ReadFile(h, p + index, count, &n, IntPtr.Zero))
            throw new Exception("bad");
    }
    return n;
}

public unsafe int Write(IntPtr h, byte[] buffer, int index, int count)
{
    int n = 0;
    fixed (byte* p = buffer) {
        if (!WriteFile(h, p + index, count, &n, IntPtr.Zero))
            throw new Exception("bad");
    }
    return n;
}

Answer 3

Raymond Chen recently covered this on his blog:

• Be careful when redirecting both a process's stdin and stdout to pipes, for you can easily deadlock
• We've traced the pipe, and it's coming from inside the process!

Answer 4

It's not very clear from your post what you mean by "I need to read it synchronously immediately after the process writes to it." If you need immediate feedback, you need asynchronous management.

Pseudocode:

Synchronous management:

string sOutput = process.StandardOutput.ReadToEnd();
process.WaitToExit();

Asynchronous management:

/*subscribe to events in order to receive notification*/    
p.StartInfo.RedirectStandardInput = true;
p.OutputDataReceived += Subscription

Afterwards if you need p.WaitForExit();, if you don't care when it finished but just want data from it, you can even avoid that line.

Hope this helps.