How to synchronize writing and async reading from interactive process's std's

Question

I am trying to build a wpf application that makes use of pythons dynamic interpreter and its eval function. EDIT: I already gave a more detailed description here In simple words, I want to be able to do something like the following:

string expression = Console.ReadLine("Please enter your expression");
if (EvaluateWithPythonProcess(expression) > 4)
{
  // Do something
}
else
{
  // Do something else
}

As my program uses this functionality during it's entire lifetime, I am not able to exit the python process each time I want to start an evaluation. As a consequence, the StdIn, StdOut and StdErr Streams remain open all the time.

I was able to start an interactive python.exe using the Process class and two corresponding OnOutputDataReceived and OnErrorDataReceived methods that transfer data from stdOut and stdErr into StringBuilders:

// create the python process StartupInfo object
                ProcessStartInfo _processStartInfo = new ProcessStartInfo(PythonHelper.PathToPython + "python.exe");

                // python uses "-i" to run in interactive mode
                _processStartInfo.Arguments = "-i";

                // Only start the python process, but don't show a (console) window
                _processStartInfo.WindowStyle = ProcessWindowStyle.Minimized;
                _processStartInfo.CreateNoWindow = true;

                // Enable the redirection of python process std's
                _processStartInfo.UseShellExecute = false;

                _processStartInfo.RedirectStandardOutput = true;
                _processStartInfo.RedirectStandardInput = true;
                _processStartInfo.RedirectStandardError = true;

                // Create the python process object and apply the startupInfos from above
                _pythonProcess = new Process();
                _pythonProcess.StartInfo = _processStartInfo;

                // Start the process, _hasStarted indicates if the process was actually started (true) or if it was reused (false, was already running)

                    _pythonProcess.OutputDataReceived += new DataReceivedEventHandler(OnOutputDataReceived);
                    _pythonProcess.ErrorDataReceived += new DataReceivedEventHandler(OnErrorDataReceived);
                    bool _hasStarted = _pythonProcess.Start();

                    _pythonProcess.BeginOutputReadLine();
                    _pythonProcess.BeginErrorReadLine();
                    _input = _pythonProcess.StandardInput;

However, I cannot manage to synchronize my application with this asynchronous gathering of results. As the two On*DataReceived() Methods are called asynchronously, I do not know if python has finished the evaluation of my expression. A possible solution would be to create a wait handle before sending commands to pythons stdIn which I can wait for afterwards. Both the OnOutputDataReceived and the OnErrorDataReceived Methods could signal this handle. However, this is somehow obscured by the intended behaviour of python:

                // example A: Import the sys modul in python 
                // this does cause neither an output, nor an error:
                _input.WriteLine("import sys");

                // example B: Writing to pythons stderr or stdout results in Error AND Output, how can I tell if an error occured?
                _input.WriteLine("sys.stderr.write('Initialized stdErr')");

                _input.WriteLine("sys.stdout.write('Initialized stdOut')");

                // example C: This is the intended use, but how can I tell if evaluation has finished succesfully?
                _input.WriteLine("print(4+7)");

                // example D: A simple typo might lead to unforeseeable errors but how can I tell if evaluation has finished succesfully?
                _input.WriteLine("pr int(4+7)");

Answer 1

I found a solution which to my mind is a workaround for my specific scenario and not a general solution.

Based on the comments from @Peter I tried to figure out how a "human would solve this problem"

1. I have to make sure that the child process communicates with parent process via stdOut only.
2. I have to create a message based protocol that ensures that the child python process does always report whether he has received and understood the message sent by the parent c# process, and if so reports the evaluated value of the expression.
3. I have to find a way to synchronize subsequent writing and reading

Points one and two are achieved by defining a python method which will always be the target for my parent-process. I make use of pythons exception handling routines to detect errors and prevent it from writing into stdErr as follows :

def PythonEval(expression):
    "Takes an expression and uses the eval function, wrapped in a try-except-statement, to inform the parent process about the value of this expression"
    try:
      print(eval(expression))
      print('CHILD: DONE')
    except:
      print('CHILD: ERROR')
    return

This definition can be applied from within my c# application by wrapping the python code in a string and passing it to the child process's stdIn:

childInput = childProcess.StandardInput;

childInput.WriteLine("def PythonEval(expression):\n" +  
    "\t\"Takes an expression and uses the eval function, wrapped in a try-except-clause, to inform the LMT about the outcome of this expression\"\n" + 
    "\ttry:\n" +
        "\t\tprint(eval(expression))\n" + 
        "\t\tprint('" + _pythonSuccessMessage + "')\n" + 
    "\texcept:\n" + 
        "\t\tprint('" + _pythonErrorMessage + "')\n" + 
    "\treturn\n" + 
    "\n"); // This last newline character is important, it ends the definition of a method if python is in interactive mode 

If I want to evaluate an expression with the help of the child process, the parent process now always has to wrap the expression in a corresponding call of this python method:

childInput.WriteLine("PythonEval('" + expression + "')");

This will in all cases result in a message to child process's stdOut which has a last line of the form "CHILD: DONE|ERROR", which I can compare to and set a boolean flag _hasError in the latter case. The entire message is passed into a stringBuilder OutputMessage.

When the child process sends this message to its stdOut, the C# process object's OutputDataReceivedEvent is fired and data is read asynchronously by the OnOutputDataReceived method. In order to synchronize with the asnyc read operations of the process, I use a AutoResetEvent. It allows both to synchronize the parent c# process with the python process, and to prevent deadlocks by using the AutoResetEvent.WaitOne(int timeout) overload.

The AutoResetEvent is reset manually in a specific method that sends the command to python, automatically after a waitOne has completed (before a timeout occured), and is set manually in the async OnOutputDataReceived() method as follows:

private AutoResetEvent _outputResetEvent;
private bool _hasError;
private StringBuilder _outputMessage;

private void EvaluateWithPython(string expression)
{
    // Set _outputResetEvent to unsignalled state
    _outputResetEvent.Reset();

    // Reset _hasError, 
    _hasError = true;

    // Write command to python, using its dedicated method
    childInput.WriteLine("PythonEval('" + expression + "')"); // The ' chars again are important, as the eval method in python takes a string, which is indicated by 's in python

    // wait for python to write into stdOut, which is captured by OnOutputDataReceived (below) and sets _outputResetEvent to signalled stat
    bool _timeoutOccured = _outputResetEvent.WaitOne(5000);

    // Give OnOutputDataReceived time to finish
    Task.Delay(200).Wait();        
}

private void OnOutputDataReceived(object sender, DataReceivedEventArgs e)
{
    if (e == null)
    {
        throw new ArgumentNullException();
    }

    if (e.Data != null)
    {
        // Pass message into StringBuilder line by line, as OnOutputDataReceived is called line by line
        _outputMessage.AppendLine(e.Data);

        // Check for end of message, this is in all cases of the form "CHILD: DONE|ERROR"
        // In this case, set errorFlag if needed and signal the AutoResetEvent
        if (e.Data.Equals("CHILD: ERROR"))
        {
            _hasError = true;
            _outputResetEvent.Set();
        }
        else if (e.Data.Equals("CHILD: DONE"))
        {
            _hasError = false;
            _outputResetEvent.Set();
        }
    }
    else
    {
        // TODO: We only reach this point if child python process ends and stdout is closed (?)
        _outputResetEvent.Set();
    }


}

With this approach I am able to call EvaluateWithPython and can synchronously:

• check if python finished before a timeout occured (and react in some way if it hadn't)
• if no timeout occured, I know that _hasError tells if evaluation was succesful
• if this is the case, outputMessage contains the result in the next-but-last line.

In order to cope with any overseen problems, I will also write an OnErrorDataReceived method, which would capture unhandled exceptions and syntax errors from the python process and could, for example, throw an exception as this should to my mind NEVER happen.