Missing Output when Redirecting Standard Error in C# / .NET Process

Question

I'm working with a 3rd party, command-line tool called "sam-ba" v3.5 (available here for free). It's a C++ / QML command line tool that interfaces with a hardware module to read/write data. Output from commands, in most cases, is sent to Standard Error.

I have a C# / .NET application that creates a Process object to execute the sam-ba tool and run commands. Executing the commands works as expected. What doesn't always work is the redirect of the Standard Error output. In some commands, part or all of the output is not received by the C# application. For example, here is the execution of a command using the sam-ba tool directly at the Windows 10 command line:

C:\Temp\Stuff\sam-ba_3.5>sam-ba -p serial:COM5 -d sama5d3 -m version
Error: Cannot open invalid port 'COM5'
Cannot open invalid port 'COM5'

Here is some simple code from a C# application to create a Process object to execute the sam-ba tool with the same command:

Process p = new Process
{
    StartInfo = new ProcessStartInfo("sam-ba.exe", "-p serial:COM5 -d sama5d3 -m version")
    {
        RedirectStandardOutput = true,
        RedirectStandardError = true,
        UseShellExecute = false,
        CreateNoWindow = true
    }
};

p.Start();
string output = p.StandardOutput.ReadToEnd();
string error = p.StandardError.ReadToEnd();
p.WaitForExit();

Console.WriteLine("Standard Out: " + output);
Console.WriteLine("Standard Error: " + error);

The Output of the C# application:

Standard Out:
Standard Error: Cannot open invalid port 'COM5'

In this simple example, only 1 of the output lines is redirected to Standard Error while the other is not. I've tried many different commands and results are mixed. Sometimes I get everything, sometimes partial output, sometimes no output.

Now ... here's the real issue. The following is a python script (v3.8) that does exactly what the C# application is doing:

import subprocess
import sys

result = subprocess.run("sam-ba.exe -p serial:COM5 -d sama5d3 -m version", capture_output=True, text=True)
print("stdout:", result.stdout)
print("stderr:", result.stderr)

This script always returns the correct output to standard error. BUT ... when I run this script from the C# app to create a chain of C# -> python -> sam-ba, I get the same issue of output missing from the stream.

This has led me to 2 conclusions:

1. Something in that sam-ba tool is different about the way it is outputting its text. Format, content, ... something. There's an inconsistency going on somewhere in that code
2. Something is different about the environment created by the C# Process object when executing external applications that doesn't happen when the external application is run directly. Otherwise, why would the python script get all the output when run directly, but not when run through the C# Process object?

It is #2 that has brought me here. I'm looking for any insight on how to diagnose this. Something I'm doing wrong, settings I can try within the Process object, thoughts on how data can go into a stream and not come out on redirect, or if anyone has ever seen something like this before and how they resolved it.

UPDATE

Got a hold of the sam-ba tool's source code. The output the C# app is not capturing is coming from the QML files. They are using this 'print()' method that I can't really find any details on. The output the C# app can capture is being delivered back to the C++ side via signals and then sent to standard error. This feeds back into my conclusion #1 where they have inconsistencies in their code.

Still, this potentially means there is a conflict between C# and QT/QML, which would explain why the Python script gets the QML output, but the C# app does not.

Answer 1

The following uses ShellExecute instead of CreateProcess when running process. When using ShellExecute one can't re-direct StandardOutput and/or StandardError for Process. To work around this, both StandardOutput and StandardError are re-directed to a temp file, and then the data is read from the temp file--which seems to result in the same output that one sees when running from a cmd window.

Note: In the following code it's necessary to use %windir%\system32\cmd.exe (ex: C:\Windows\system32\cmd.exe) with the /c option. See the usage section below.

Add using statement: using System.Diagnostics;

Then try the following:

public string RunProcess(string fqExePath, string arguments, bool runAsAdministrator = false)
{
    string result = string.Empty;
    string tempFilename = System.IO.Path.Combine(System.IO.Path.GetTempPath(), "tempSam-ba.txt");
    string tempArguments = arguments;

    if (String.IsNullOrEmpty(fqExePath))
    {
        Debug.WriteLine("fqExePath not specified");
        return "Error: fqExePath not specified";
    }

    //redirect both StandardOutput and StandardError to a temp file
    if (!arguments.Contains("2>&1"))
    {
        tempArguments += String.Format(" {0} {1} {2}", @"1>", tempFilename, @"2>&1");
    }

    //create new instance
    ProcessStartInfo startInfo = new ProcessStartInfo(fqExePath, tempArguments);

    if (runAsAdministrator)
    {
        startInfo.Verb = "runas"; //elevates permissions
    }//if

    //set environment variables
    //pStartInfo.EnvironmentVariables["SomeVar"] = "someValue";

    startInfo.RedirectStandardError = false;
    startInfo.RedirectStandardOutput = false;

    startInfo.RedirectStandardInput = false;

    startInfo.UseShellExecute = true; //use ShellExecute instead of CreateProcess
    startInfo.CreateNoWindow = false;

    startInfo.WindowStyle = ProcessWindowStyle.Hidden;
    startInfo.ErrorDialog = false;
    startInfo.WorkingDirectory = System.IO.Path.GetDirectoryName(fqExePath);

    using (Process p = Process.Start(startInfo))
    {
        //start
        p.Start();

        //waits until the process is finished before continuing
        p.WaitForExit();
    }

    //read output from temp file
    //file may still be in use, so try to read it.
    //if it is still in use, sleep and try again
    if (System.IO.File.Exists(tempFilename))
    {
        string errMsg = string.Empty;
        int count = 0;
        do
        {
            //re-initialize
            errMsg = string.Empty;

            try
            {
                result = System.IO.File.ReadAllText(tempFilename);
                Debug.WriteLine(result);
            }
            catch(System.IO.IOException ex)
            {
                errMsg = ex.Message;
            }
            catch (Exception ex)
            {
                errMsg = ex.Message;
            }

            System.Threading.Thread.Sleep(125);
            count += 1; //increment
        } while (!String.IsNullOrEmpty(errMsg) && count < 10);

        //delete temp file
        System.IO.File.Delete(tempFilename);
    }

    return result;
}

Usage:

RunProcess(@"C:\Windows\system32\cmd.exe", @"/c C:\Temp\sam-ba_3.5\sam-ba.exe -p serial:COM5 -d sama5d3 -m version");

Note: /c C:\Temp\sam-ba_3.5\sam-ba.exe -p serial:COM5 -d sama5d3 -m version is the value of the process "Argument" property.

Update:

Option 2:

Here's a solution that uses named pipes. Process is used to redirect the output to a named pipe instead of a file. One creates a named pipe "server" which listens for a connection from a client.Then System.Diagnostics.Process is used to run the desired command and redirect the output to the named pipe server. The "server" reads the output, and then raises event "DataReceived" which will return the data to any subscribers.

The named pipe server code is from here, however I've modified it. I've added numerous events--which can be subscribed to. I've also added the ability for the server to shut itself down after it's finished reading the data by setting "ShutdownWhenOperationComplete" to "true".

Create a class named: HelperNamedPipeServer.cs

HelperNamedPipeServer.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO.Pipes;
using System.IO;
using System.Diagnostics;
using System.Threading;
using System.Security.Principal;

namespace ProcessTest
{
    public class HelperNamedPipeServer : IDisposable
    {
        //delegates
        public delegate void EventHandlerClientConnected(object sender, bool e);
        public delegate void EventHandlerDataReceived(object sender, string data);
        public delegate void EventHandlerOperationCompleted(object sender, bool e);
        public delegate void EventHandlerMessageComplete(object sender, bool e);
        public delegate void EventHandlerReadComplete(object sender, bool e);
        public delegate void EventHandlerServerShutdown(object sender, bool e);
        public delegate void EventHandlerServerStarted(object sender, bool e);

        //event that subscribers can subscribe to
        public event EventHandlerClientConnected ClientConnected;
        public event EventHandlerDataReceived DataReceived;
        public event EventHandlerMessageComplete MessageReadComplete;
        public event EventHandlerOperationCompleted OperationCompleted;
        public event EventHandlerReadComplete ReadComplete;
        public event EventHandlerServerShutdown ServerShutdown;
        public event EventHandlerServerStarted ServerStarted;

        
        public bool IsClientConnected
        {
            get
            {
                if (_pipeServer == null)
                {
                    return false;
                }
                else
                {
                    return _pipeServer.IsConnected;
                }
            }
        }

        public string PipeName { get; set; } = string.Empty;

        public bool ShutdownWhenOperationComplete { get; set; } = false;

        //private int _bufferSize = 4096;
        private int _bufferSize = 65535;

        //private volatile NamedPipeServerStream _pipeServer = null;
        private NamedPipeServerStream _pipeServer = null;

        public HelperNamedPipeServer()
        {
            PipeName = "sam-ba-pipe";
        }

        public HelperNamedPipeServer(string pipeName)
        {
            PipeName = pipeName;
        }

        private NamedPipeServerStream CreateNamedPipeServerStream(string pipeName)
        {
            //named pipe with security
            //SecurityIdentifier sid = new SecurityIdentifier(WellKnownSidType.BuiltinAdministratorsSid, null); //member of Administrators group
            //SecurityIdentifier sid = new SecurityIdentifier(WellKnownSidType.WorldSid, null); //everyone
            //SecurityIdentifier sid = new SecurityIdentifier(WellKnownSidType.BuiltinUsersSid, null); //member of Users group

            //PipeAccessRule rule = new PipeAccessRule(sid, PipeAccessRights.ReadWrite, System.Security.AccessControl.AccessControlType.Allow);
            //PipeSecurity pSec = new PipeSecurity();
            //pSec.AddAccessRule(rule);

            //named pipe - with specified security
            //return new NamedPipeServerStream(PipeName, PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Byte, PipeOptions.Asynchronous, _bufferSize, _bufferSize, pSec);

            //named pipe - access for everyone
            //return new System.IO.Pipes.NamedPipeServerStream(pipeName, PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Message, PipeOptions.Asynchronous);
            return new System.IO.Pipes.NamedPipeServerStream(pipeName, PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);
            
        }

        public void Dispose()
        {
            Shutdown();
        }

        private void OnClientConnected()
        {
            LogMsg("OnClientConnected");

            //raise event
            if (ClientConnected != null)
                ClientConnected(this, true);
        }

        private void OnDataReceived(string data)
        {
            LogMsg("OnClientConnected");

            //raise event
            if (DataReceived != null && !String.IsNullOrEmpty(data))
            {
                if (DataReceived != null)
                    DataReceived(this, data);
            }
        }

        private void OnMessageReadComplete()
        {
            LogMsg("OnMessageReadComplete");

            //raise event
            if (MessageReadComplete != null)
                MessageReadComplete(this, true);
        }


        private void OnOperationCompleted()
        {
            LogMsg("OnOperationCompleted");

            //raise event
            if (OperationCompleted != null)
                OperationCompleted(this, true);
        }

        private void OnReadComplete()
        {
            LogMsg("OnReadComplete");

            //raise event
            if (ReadComplete != null)
                ReadComplete(this, true);
        }

        private void OnServerShutdown()
        {
            LogMsg("OnServerShutdown");

            //raise event
            if (ServerShutdown != null)
                ServerShutdown(this, true);
        }

        private void OnServerStarted()
        {
            LogMsg("OnServerStarted");

            //raise event
            if (ServerStarted != null)
                ServerStarted(this, true);
        }


        private async void DoConnectionLoop(IAsyncResult result)
        {   //wait for connection, then process the data

            if (!result.IsCompleted) return;
            if (_pipeServer == null) return;

            //IOException = pipe is broken
            //ObjectDisposedException = cannot access closed pipe
            //OperationCanceledException - read was canceled

            //accept client connection
            try
            {
                //client connected - stop waiting for connection
                _pipeServer.EndWaitForConnection(result);

                OnClientConnected(); //raise event
            }
            catch (IOException) { RebuildNamedPipe(); return; }
            catch (ObjectDisposedException) { RebuildNamedPipe(); return; }
            catch (OperationCanceledException) { RebuildNamedPipe(); return; }

            while (IsClientConnected)
            {
                if (_pipeServer == null) break;

                try
                {
                    // read from client
                    string clientMessage = await ReadClientMessageAsync(_pipeServer);

                    OnDataReceived(clientMessage); //raise event
                }
                catch (IOException) { RebuildNamedPipe(); return; }
                catch (ObjectDisposedException) { RebuildNamedPipe(); return; }
                catch (OperationCanceledException) { RebuildNamedPipe(); return; }
            }

            //raise event
            OnOperationCompleted();

            if (!ShutdownWhenOperationComplete)
            {
                
                //client disconnected. start listening for clients again
                if (_pipeServer != null)
                    RebuildNamedPipe();
            }
            else
            {
                Shutdown();
            }
        }

        private void LogMsg(string msg)
        {
            //ToDo: log message
            string output = String.Format("{0} - {1}", DateTime.Now.ToString("yyyy/MM/dd HH:mm:ss"), msg);

            //ToDo: uncomment this line, if desired
            //Debug.WriteLine(output);
        }

        private void RebuildNamedPipe()
        {
            Shutdown();
            _pipeServer = CreateNamedPipeServerStream(PipeName);
            _pipeServer.BeginWaitForConnection(DoConnectionLoop, null);
        }


        private async Task<string> ReadClientMessageAsync(NamedPipeServerStream stream)
        {
            byte[] buffer = null;
            string clientMsg = string.Empty;
            StringBuilder sb = new StringBuilder();
            int msgIndex = 0;
            int read = 0;

            LogMsg("Reading message...");

            if (stream.ReadMode == PipeTransmissionMode.Byte)
            {
                LogMsg("PipeTransmissionMode.Byte");

                //byte mode ignores message boundaries
                do
                {
                    //create instance
                    buffer = new byte[_bufferSize];

                    read = await stream.ReadAsync(buffer, 0, buffer.Length);

                    if (read > 0)
                    {
                        clientMsg = Encoding.UTF8.GetString(buffer, 0, read);
                        //string clientMsg = Encoding.Default.GetString(buffer, 0, read);

                        //remove newline
                        //clientMsg = System.Text.RegularExpressions.Regex.Replace(clientString, @"\r\n|\t|\n|\r|", "");

                        //LogMsg("clientMsg [" + msgIndex + "]: " + clientMsg);
                        sb.Append(clientMsg);

                        msgIndex += 1; //increment
                    }
                } while (read > 0);

                //raise event
                OnReadComplete();
                OnMessageReadComplete();
            }
            else if (stream.ReadMode == PipeTransmissionMode.Message)
            {
                LogMsg("PipeTransmissionMode.Message");

                do
                {
                    do
                    {
                        //create instance
                        buffer = new byte[_bufferSize];

                        read = await stream.ReadAsync(buffer, 0, buffer.Length);

                        if (read > 0)
                        {
                            clientMsg = Encoding.UTF8.GetString(buffer, 0, read);
                            //string clientMsg = Encoding.Default.GetString(buffer, 0, read);

                            //remove newline
                            //clientMsg = System.Text.RegularExpressions.Regex.Replace(clientString, @"\r\n|\t|\n|\r|", "");

                            //LogMsg("clientMsg [" + msgIndex + "]: " + clientMsg);
                            sb.Append(clientMsg);

                            msgIndex += 1; //increment
                        }
                    } while (!stream.IsMessageComplete);

                    //raise event
                    OnMessageReadComplete();
                } while (read > 0);

                //raise event
                OnReadComplete();

                LogMsg("message completed");
            }

            return sb.ToString();
        }

        private void Shutdown()
        {
            LogMsg("Shutting down named pipe server");

            if (_pipeServer != null)
            {
                try { _pipeServer.Close(); } catch { }
                try { _pipeServer.Dispose(); } catch { }
                _pipeServer = null;
            }
        }

        public void StartServer(object obj = null)
        {
            LogMsg("Info: Starting named pipe server...");

            _pipeServer = CreateNamedPipeServerStream(PipeName);
            _pipeServer.BeginWaitForConnection(DoConnectionLoop, null);
        }

        public void StopServer()
        {
            Shutdown();
            OnServerShutdown(); //raise event
            LogMsg("Info: Server shutdown.");
        }
    }
}

Next, I've created a "Helper" class that contains the code to start the named pipe server, run the command using Process, and return the data. There are three ways to get the data. It's returned by the method, one can subscribe to the "DataReceived" event, or once the method completes, the data will be in property "Data".

Helper.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.IO.Pipes;
using System.IO;
using System.Threading;

namespace ProcessTest
{
    public class Helper : IDisposable
    {

        public delegate void EventHandlerDataReceived(object sender, string data);

        //event that subscribers can subscribe to
        public event EventHandlerDataReceived DataReceived;

        private StringBuilder _sbData = new StringBuilder();

        private HelperNamedPipeServer _helperNamedPipeServer = null;

        private bool _namedPipeServerOperationComplete = false;

        public string Data { get; private set; } = string.Empty;

        public Helper()
        {

        }

        private void OnDataReceived(string data)
        {
            if (!String.IsNullOrEmpty(data) && DataReceived != null)
            {
                DataReceived(this, data);

                //Debug.Write("Data: " + data);
            }
        }

        public void Dispose()
        {
            ShutdownNamedPipeServer();
        }

        public async Task<string> RunSambaNamedPipesAsync(string fqExePath, string arguments, string pipeName = "sam-ba-pipe", string serverName = ".", bool runAsAdministrator = false)
        {
            string result = string.Empty;
            string tempArguments = arguments;

            //re-initialize
            _namedPipeServerOperationComplete = false;
            _sbData = new StringBuilder();
            Data = string.Empty;

            if (String.IsNullOrEmpty(fqExePath))
            {
                Debug.WriteLine("fqExePath not specified");
                return "fqExePath not specified";
            }

            //create new instance
            _helperNamedPipeServer = new HelperNamedPipeServer(pipeName);
            _helperNamedPipeServer.ShutdownWhenOperationComplete = true;

            //subscribe to events
            _helperNamedPipeServer.DataReceived += HelperNamedPipeServer_DataReceived;
            _helperNamedPipeServer.OperationCompleted += HelperNamedPipeServer_OperationCompleted;

            //start named pipe server on it's own thread
            Thread t = new Thread(_helperNamedPipeServer.StartServer);
            t.Start();

            //get pipe name to use with Process
            //this is where output from the process
            //will be redirected to
            string fqNamedPipe = string.Empty;

            if (String.IsNullOrEmpty(serverName))
            {
                fqNamedPipe = String.Format(@"\\{0}\pipe\{1}", serverName, pipeName);
            }
            else
            {
                fqNamedPipe = String.Format(@"\\{0}\pipe\{1}", ".", pipeName);
            }

            //redirect both StandardOutput and StandardError to named pipe
            if (!arguments.Contains("2>&1"))
            {
                tempArguments += String.Format(" {0} {1} {2}", @"1>", fqNamedPipe, @"2>&1");
            }

            //run Process
            RunProcess(fqExePath, tempArguments, runAsAdministrator);

            while (!_namedPipeServerOperationComplete)
            {
                await Task.Delay(125);
            }

            //set value
            Data = _sbData.ToString();

            return Data;

        }


        public void RunProcess(string fqExePath, string arguments,  bool runAsAdministrator = false)
        {

            if (String.IsNullOrEmpty(fqExePath))
            {
                Debug.WriteLine("fqExePath not specified");
                throw new Exception( "Error: fqExePath not specified");
            }

            //create new instance
            ProcessStartInfo startInfo = new ProcessStartInfo(fqExePath, arguments);

            if (runAsAdministrator)
            {
                startInfo.Verb = "runas"; //elevates permissions
            }//if

            //set environment variables
            //pStartInfo.EnvironmentVariables["SomeVar"] = "someValue";

            startInfo.RedirectStandardError = false;
            startInfo.RedirectStandardOutput = false;

            startInfo.RedirectStandardInput = false;

            startInfo.UseShellExecute = true; //use ShellExecute instead of CreateProcess

            startInfo.WindowStyle = ProcessWindowStyle.Hidden;
            startInfo.ErrorDialog = false;
            startInfo.WorkingDirectory = System.IO.Path.GetDirectoryName(fqExePath);

            using (Process p = Process.Start(startInfo))
            {
                //start
                p.Start();

                //waits until the process is finished before continuing
                p.WaitForExit();
            }
        }

        private void HelperNamedPipeServer_OperationCompleted(object sender, bool e)
        {
            //Debug.WriteLine("Info: Named pipe server - Operation completed.");

            //set value
            Data = _sbData.ToString();

            //set value
            _namedPipeServerOperationComplete = true;

        }

        private void HelperNamedPipeServer_DataReceived(object sender, string data)
        {
            Debug.WriteLine("Info: Data received from named pipe server.");

            if (!String.IsNullOrEmpty(data))
            {
                //append
                _sbData.Append(data.TrimEnd('\0'));

                //send data to subscribers
                OnDataReceived(data);
            }
        }

        private void ShutdownNamedPipeServer()
        {
            Debug.WriteLine("Info: ShutdownNamedPipeServer");
            try
            {
                if (_helperNamedPipeServer != null)
                {
                    //unsubscribe from events
                    _helperNamedPipeServer.DataReceived -= HelperNamedPipeServer_DataReceived;
                    _helperNamedPipeServer.OperationCompleted -= HelperNamedPipeServer_OperationCompleted;

                    _helperNamedPipeServer.Dispose();
                    _helperNamedPipeServer = null;
                }
            }
            catch (Exception ex)
            {
            }
        }
    }
}

Usage:

private async void btnRunUsingNamedPipes_Click(object sender, EventArgs e)
{
    //Button name: btnRunUsingNamedPipes

    using (Helper helper = new Helper())
    {
        //subscribe to event
        helper.DataReceived += Helper_DataReceived;

        var result = await helper.RunSambaNamedPipesAsync(@"C:\Windows\system32\cmd.exe", @"/c C:\Temp\sam-ba_3.5\sam-ba.exe -p serial:COM5 -d sama5d3 -m version");
        Debug.WriteLine("Result: " + result);

        //unsubscribe from event
        helper.DataReceived -= Helper_DataReceived;
    }
}

private void Helper_DataReceived(object sender, string data)
{
    //System.Diagnostics.Debug.WriteLine(data);

    //RichTextBox name: richTextBoxOutput

    if (richTextBoxOutput.InvokeRequired)
    {
        richTextBoxOutput.Invoke((MethodInvoker)delegate
        {
            richTextBoxOutput.Text = data;
            richTextBoxOutput.Refresh();
        });
    }
}

Resources:

• When do we need to set ProcessStartInfo.UseShellExecute to True?
• Redirecting error messages from Command Prompt: STDERR/STDOUT
• PipeTransmissionMode.Message: How do .NET named pipes distinguish between messages?
• WellKnownSidType Enum