Why does BufReader hang when reading from stderr?

Question

I want to execute a command and then capture any potential output to stderr. Here's what I have:

if let Ok(ref mut child) = Command::new("ssh")
            .args(&[
              "some_args",
              "more_args"
            ])
            .stderr(Stdio::piped())
            .spawn()
        {
            let output = child.wait().expect("ssh command not running");
            let reader = BufReader::new(child.stderr.take().expect("failed to capture stderr"));
            for line in reader.lines() {
                match line {
                    Ok(line_str) => println!("output: {}", line_str);
                    Err(e) => println!("output failed!"),
                }
            }
        }

I see the output being printed but the program then hangs. I'm suspecting that this may be related to the child process exiting and BufReader is unable to read an eof. A work around was to maintain an let mut num_lines = 0; and then increment this per read. After x-number of reads, I break in the for-loop but this doesn't seem very clean. How can I get BufReader to finish reading properly?

Answer 1

Neither of these may solve your issue, but I'll offer the advice regardless:

Pipe-Wait-Read can deadlock

Calling child.wait() will block execution until the child has exited, returning the exit status.

Using Stdio::piped() creates a new pipe for the stdout/stderr streams in order to be processed by the application. Pipes are handled by the operating system and are not infinite; if one end of the pipe is writing data but the other side isn't reading it, it will eventually block those writes until something is read.

This code can deadlock because you're waiting on the child process to exit, but it may not be able to if it becomes blocked trying to write to an output pipe thats full and not being read from.

As an example, this deadlocks on my system (a fairly standard ubuntu system that has 64KiB buffers for pipes):

// create a simple child proccess that sends 64KiB+1 random bytes to stdout
let mut child = Command::new("dd")
    .args(&["if=/dev/urandom", "count=65537", "bs=1", "status=none"])
    .stdout(Stdio::piped())
    .spawn()
    .expect("failed to execute dd");

let _status = child.wait(); // hangs indefinitely
let reader = BufReader::new(child.stdout.take().expect("failed to capture stdout"));
for _line in reader.lines() {
    // do something
}

There are plenty of alternatives:

• Just read the output without waiting. reader.lines() will stop iterating when it reaches the end of the stream. You can then call child.wait() if you want to know the exit status.

• Use .output() instead of .spawn(). This will block until the child has exited and return an Output holding the full stdout/stderr streams as Vec<u8>s.

• You can process the output streams in separate threads while you're waiting for the child to exit. If that sounds good consider using tokio::process::Command.

See How do I read the output of a child process without blocking in Rust? for more info.

Don't swallow errors from .lines()

reader.lines() returns an iterator that yields a result for each line. One of the error states that could be somewhat handled is if the line wasn't properly utf-8 encoded, which will return something like this:

Err(
    Custom {
        kind: InvalidData,
        error: "stream did not contain valid UTF-8",
    },
)

However, any other error would be directly from the underlying reader and you should probably not continue iterating. Any error you receive is unlikely to be recoverable, and certainly not by continuing to ask for more lines.