Read from process stdout into memory stream asynchronously

Question

UPDATE: The key part is that the stream we are using is too large for the buffer used by copyto\copytoasync and so rather than running the whole task asynchronously, we need to asynchronously process the stream in parts, like the question linked to below

Please excuse the VB.net code here. I also speak C# so feel free to repsond in either language.

I have been following the example here ProcessStartInfo hanging on "WaitForExit"? Why? in order to try and solve a problem we are having running out of buffer space

Previously we were trying this code:

        Dim buffer As Byte() = New Byte(32767) {}
        Dim file As Byte()
        Using ms = New MemoryStream()
            While True
                Dim read As Integer = Process.StandardOutput.BaseStream.Read(buffer, 0, buffer.Length)

                If read <= 0 Then
                    Exit While
                End If
                ms.Write(buffer, 0, read)
            End While
            file = ms.ToArray()
        End Using

        If Not Process.WaitForExit(timeOut) Then
            Throw New Exception("Html to PDF conversion timed out.")
        End If

Now I have started to convert this to the aynch approach from the linked question, but am having trouble writing to a memory stream rather than a stringbuilder. This is what I have got so far:

Dim output = new MemoryStream()
Dim errorOutput = new StringBuilder()

Using process = New Process()

    Using outputWaitHandle As New AutoResetEvent(False)
        Using errorWaitHandle As New AutoResetEvent(False)

            process.OutputDataReceived = Function(sender, e)
                If e.Data Is Nothing Then
                    outputWaitHandle.Set()
                Else
                    output.Write(e.Data); //compile error here
                End If

            End Function

        End Using
    End Using

End Using

Of course e.Data is a string, but not only that I also need a buffer and an offset... and not sure what to provide here.

Any suggestions welcome, thanks!

score 1 · Answer 1 · answered Nov 30 '12 at 15:09

1

Instead of using OutputDataReceived, you can use the stream directly:

static async Task<MemoryStream> ReadProcessOutput(ProcessStartInfo psi)
{
    MemoryStream ms = new MemoryStream();

    using (Process p = new Process())
    {
        p.StartInfo = psi;

        TaskCompletionSource<int> tcs = new TaskCompletionSource<int>();
        EventHandler eh = (s, e) => tcs.TrySetResult(0);

        p.Exited += eh;

        try
        {
            p.EnableRaisingEvents = true;
            p.Start();

            await p.StandardError.BaseStream.CopyToAsync(ms);
            await tcs.Task;
        }
        finally
        {
            p.Exited -= eh;
        }
    }

    return ms;
}

answered Nov 30 '12 at 15:09

Cory Nelson

29,236
5
72
110

Thanks for this... there is nothing wrong with this answer, except that now I just have the same problem but running on a seperate thread! The key part is that the stream we are using is too large for the buffer used by copyto\copytoasync and so rather than running the whole task asynchronously, we need to asynchronously process the stream in parts, like the example linked to in the question. – beyond-code Nov 30 '12 at 16:59
Ahhhh, ok, I understand now. Yes, instead of CopyToAsync can you just keep ReadAsync()ing and process in chunks? It's not very clear if you need a specific chunk size, if it is determined from input, or if it doesn't matter? – Cory Nelson Nov 30 '12 at 17:37

score 0 · Answer 2 · answered Nov 30 '12 at 15:11

You may want to look into using the StreamWriter class instead of just a MemoryStream. That would allow you to do something like this:

MemoryStream output = new MemoryStream();
StreamWriter outputWriter = new StreamWriter(output);

...[snip]...

outputWriter.Write(e.Data);

Read from process stdout into memory stream asynchronously

2 Answers2