capture process stdout and stderr in the correct ordering

Question

I launch a process from C# as follows:

public bool Execute()
{
    ProcessStartInfo startInfo = new ProcessStartInfo();

    startInfo.Arguments =  "the command";
    startInfo.FileName = "C:\\MyApp.exe";

    startInfo.UseShellExecute = false;
    startInfo.RedirectStandardOutput = true;
    startInfo.RedirectStandardError = true;

    Log.LogMessage("{0} {1}", startInfo.FileName, startInfo.Arguments);

    using (Process myProcess = Process.Start(startInfo))
    {
        StringBuilder output = new StringBuilder();
        myProcess.OutputDataReceived += delegate(object sender, DataReceivedEventArgs e)
        {
            Log.LogMessage(Thread.CurrentThread.ManagedThreadId.ToString() + e.Data);
        };
        myProcess.ErrorDataReceived += delegate(object sender, DataReceivedEventArgs e)
        {
            Log.LogError(Thread.CurrentThread.ManagedThreadId.ToString() +  " " + e.Data);            
        };

        myProcess.BeginErrorReadLine();
        myProcess.BeginOutputReadLine();

        myProcess.WaitForExit();

    }

    return false;
}

But this has a problem... if the app in question writes to std out and std err in this order:

std out: msg 1
std err: msg 2
std out: msg 3

Then the output I see from the logs is:

msg 2
msg 1
msg 3

This seems to be because the event handlers are executed in another thread. So my question is how can the order of the process writing to std err and std out be maintained?

I thought of using a time stamp but I don't think this will work due to the preemptive nature of threads..

Update: Confirmed that using a time stamp on the data is no use.

Final update: The accepted answer solves this problem - however it does have one drawback, when the streams are merged there is no way to know which stream was written to. Hence if you require the logic of write to stderr == failure rather than the app exit code you might still be screwed.

Answer 1

As far I understand, you want to preserve the order of stdout/stderr messages. I don't see any DECENT way to do this with C# managed Process(reflection - yes, nasty subclassing hacking - yes). It seems that it's pretty much hardcoded.

This functionality does not depend on threads themselves. If you want to keep the order, STDOUT and STDERROR have to use same handle(buffer). If they use the same buffer, it's going to be synchronized.

Here is a snippet from Process.cs:

 if (startInfo.RedirectStandardOutput) {
    CreatePipe(out standardOutputReadPipeHandle, 
               out startupInfo.hStdOutput, 
               false);
    } else {
    startupInfo.hStdOutput = new SafeFileHandle(
         NativeMethods.GetStdHandle(
                         NativeMethods.STD_OUTPUT_HANDLE), 
                         false);
}

if (startInfo.RedirectStandardError) {
    CreatePipe(out standardErrorReadPipeHandle, 
               out startupInfo.hStdError, 
               false);
    } else {
    startupInfo.hStdError = new SafeFileHandle(
         NativeMethods.GetStdHandle(
                         NativeMethods.STD_ERROR_HANDLE),
                         false);
}

as you can see, there are gonna be two buffers, and if we have two buffers, we have already lost the order information.

Basically, you need to create your own Process() class that can handle this case. Sad? Yes. The good news is that it's not hard, it seems pretty simple. Here is a code taken from StackOverflow, not C# but enough to understand the algorithm:

function StartProcessWithRedirectedOutput(const ACommandLine: string; const AOutputFile: string;
  AShowWindow: boolean = True; AWaitForFinish: boolean = False): Integer;
var
  CommandLine: string;
  StartupInfo: TStartupInfo;
  ProcessInformation: TProcessInformation;
  StdOutFileHandle: THandle;
begin
  Result := 0;

  StdOutFileHandle := CreateFile(PChar(AOutputFile), GENERIC_WRITE, FILE_SHARE_READ, nil, CREATE_ALWAYS,
    FILE_ATTRIBUTE_NORMAL, 0);
  Win32Check(StdOutFileHandle <> INVALID_HANDLE_VALUE);
  try
    Win32Check(SetHandleInformation(StdOutFileHandle, HANDLE_FLAG_INHERIT, 1));
    FillChar(StartupInfo, SizeOf(TStartupInfo), 0);
    FillChar(ProcessInformation, SizeOf(TProcessInformation), 0);

    StartupInfo.cb := SizeOf(TStartupInfo);
    StartupInfo.dwFlags := StartupInfo.dwFlags or STARTF_USESTDHANDLES;
    StartupInfo.hStdInput := GetStdHandle(STD_INPUT_HANDLE);
    StartupInfo.hStdOutput := StdOutFileHandle;
    StartupInfo.hStdError := StdOutFileHandle;

    if not(AShowWindow) then
    begin
      StartupInfo.dwFlags := StartupInfo.dwFlags or STARTF_USESHOWWINDOW;
      StartupInfo.wShowWindow := SW_HIDE;
    end;

    CommandLine := ACommandLine;
    UniqueString(CommandLine);

    Win32Check(CreateProcess(nil, PChar(CommandLine), nil, nil, True,
      CREATE_NEW_PROCESS_GROUP + NORMAL_PRIORITY_CLASS, nil, nil, StartupInfo, ProcessInformation));

    try
      Result := ProcessInformation.dwProcessId;

      if AWaitForFinish then
        WaitForSingleObject(ProcessInformation.hProcess, INFINITE);

    finally
      CloseHandle(ProcessInformation.hProcess);
      CloseHandle(ProcessInformation.hThread);
    end;

  finally
    CloseHandle(StdOutFileHandle);
  end;
end;

Source: How to redirect large amount of output from command executed by CreateProcess?

Instead of file, you want to use CreatePipe. From pipe, you can read asynchronously like so:

standardOutput = new StreamReader(new FileStream(
                       standardOutputReadPipeHandle, 
                       FileAccess.Read, 
                       4096, 
                       false),
                 enc, 
                 true, 
                 4096);

and BeginReadOutput()

  if (output == null) {
        Stream s = standardOutput.BaseStream;
        output = new AsyncStreamReader(this, s, 
          new UserCallBack(this.OutputReadNotifyUser), 
             standardOutput.CurrentEncoding);
    }
    output.BeginReadLine();

Answer 2

While I appreciate Erti-Chris's answer (what is that, Pascal?), I thought others might prefer an answer in a managed language. Also, to the detractors who say that "you shouldn't be doing this" because STDOUT and STDERR are not guaranteed to preserve the ordering: yes, I understand, but sometimes we have to interoperate with programs (that we did not write) that expect us to do just that, correct semantics be damned.

Here's a version in C#. Instead of circumventing the managed Process API by calling CreateProcess, it uses an alternative approach that redirects STDERR onto the STDOUT stream in the Windows shell. Because UseShellExecute = true does not actually use the cmd.exe shell (surprise!), you normally cannot use shell redirects. The workaround is to launch the cmd.exe shell ourselves, feeding our real shell program and arguments to it manually.

Note that the following solution assumes that your args array is already properly escaped. I like the brute force solution of using the kernel's GetShortPathName call, but you should know that it is not always appropriate to use (like if you're not on NTFS). Also, you really do want to go the extra step of reading the STDOUT buffer asynchronously (as I do below), because if you don't, your program may deadlock.

using System;
using System.Diagnostics;
using System.Text;
using System.Threading;

public static string runCommand(string cpath, string[] args)
{
    using (var p = new Process())
    {
        // notice that we're using the Windows shell here and the unix-y 2>&1
        p.StartInfo.FileName = @"c:\windows\system32\cmd.exe";
        p.StartInfo.Arguments = "/c \"" + cpath + " " + String.Join(" ", args) + "\" 2>&1";
        p.StartInfo.UseShellExecute = false;
        p.StartInfo.RedirectStandardOutput = true;
        p.StartInfo.RedirectStandardError = true;

        var output = new StringBuilder();

        using (var outputWaitHandle = new AutoResetEvent(false))
        {
            p.OutputDataReceived += (sender, e) =>
            {
                // attach event handler
                if (e.Data == null)
                {
                    outputWaitHandle.Set();
                }
                else
                {
                    output.AppendLine(e.Data);
                }
            };

            // start process
            p.Start();

            // begin async read
            p.BeginOutputReadLine();

            // wait for process to terminate
            p.WaitForExit();

            // wait on handle
            outputWaitHandle.WaitOne();

            // check exit code
            if (p.ExitCode == 0)
            {
                return output.ToString();
            }
            else
            {
                throw new Exception("Something bad happened");
            }
        }
    }
}