Read portion of lines from child process in Rust (chunk of data)

Question

When I try to spawn a child ffmpeg process I use additonal flag -progress, next I use pipe to pass this progress output to the stderr. So the whole command looks like:

ffmpeg -i ... -progress pipe:2 ...

Without -progress flag ffmepg outputs following line in stderr, probably once per second:

frame=46 46 fps=0.0 q=0.0 size=       0kB time=00:00:01.72 bitrate=   0.2kbits/s speed=2.69x

With -progress flag ffmepg outputs (multiple lines) in stderr, probably once per second:

frame=1   1 fps=0.0 q=0.0 size=       0kB time=00:00:00.19 bitrate=   2.0kbits/s speed=2.94x    
fps=0.00
stream_0_0_q=0.0
bitrate=   2.0kbits/s
total_size=48
out_time_us=192000
out_time_ms=192000
out_time=00:00:00.192000
dup_frames=0
drop_frames=0
speed=2.94x
progress=continue

The main puppose of using -progress flag is to calc percentage of completion by parsing out_time_ms line and comparing to the whole duration.

Reading this chunk (portion of lines) is pretty simple in NodeJS:

const { spawn } = require('child_process');
const child = spawn('ffmpeg', [..., '-progress', 'pipe:2', ...]);

child.stderr.on('data', (data) => {
  // data will contain multiple lines, exactly one chunk
});

Reading this chunk (portion of lines) is pretty simple in Deno also:

const child = Deno.spawnChild("ffmpeg", {
  args: [..., '-progress', 'pipe:2', ...],
});
const stderrReader = child.stderr.getReader();
let readResult = await stderrReader.read();
while (readResult.done !== true) {
  readResult = await stderrReader.read();
  // readResult will contain multiple lines, exactly one chunk
}

I can't achieve the same in rust:

let mut command = Command::new("ffmpeg");
command.args(["...", "-progress", "pipe:2", "..."]);
let mut child = command
  .stdout(Stdio::piped())
  .stderr(Stdio::piped())
  .spawn()
  .unwrap();

let child_stderr = child.stderr.as_mut().expect("Unable to pipe stderr");
let mut reader = BufReader::new(child_stderr);
let mut buff = String::new();
while reader.read_line(&mut buff).expect("Unable to read chunk") > 0 {
  // buff will contain only on line
  buff.clear();
}

I am new in Rust. I can not detect what character signals end of chunk.

Runnig read_line() - will read only one line.
Runnig read_to_end() - will read the whole output until the end of process (EOF).

How can I read in Rust portion of lines that ffmpeg outputs probably once per second? How Node/Deno detects this "end of chunk"? Does rust have such event/signal also?

Your `Deno` code isn't reliable and depends on the buffer behavior of the OS. You could receive the output in multiple chunks, or combined with other things. Same in NodeJS. The only reason you receive those as one big chunk is because the OS buffers them and delivers them in a single packet, for performance reasons. — Finomnis, Oct 01 '22 at 18:11
After reading about `-progress`, I don't think it was ever meant to be used as a chunk of data. Instead, I think it should be read line by line, and every line should be evaluated separately. — Finomnis, Oct 01 '22 at 18:14
To get the exact same behaviour as in the other languages, don't use `BufReader` but instead read directly from `child_stderr` via [`child_stderr.read()`](https://doc.rust-lang.org/std/process/struct.ChildStderr.html#method.read). This will give you the 'chunking' behaviour you are talking about. But be warned, again, that **THIS IS NOT RELIABLE IN ANY LANGUAGE**. Stderr/out is meant to be processed as a continuous byte stream, and that's why `BufReader` exists. — Finomnis, Oct 01 '22 at 18:17
Thanks @Finomnis that explains much! So Node/Deno uses OS buffer and low level Rust-code showed this. — Alexey Vol, Oct 01 '22 at 18:26
Just to clarify, not sure if I made that point clear: `read()` is not erroneous or dangerous or anything like this. It's just that the start/end of the delivered chunks have to be considered arbitrarily chosen. In your case, they just happen to line up *most* of the times because the data gets produced in a very short time frame. But there is nothing that makes sure that the OS doesn't arbitrarily decide to resize the buffer to say 32-bytes, for example. Then you could theoretically receive it in a bunch of 32-byte pieces. — Finomnis, Oct 01 '22 at 19:09

Finomnis · Accepted Answer · 2022-10-02T10:52:53.287

You can use ChildStderr::read to achieve the same behaviour as in the other languages.

⚠ WARNING ⚠
This behaviour is not reliable, in any language.

Stderr is a stream of data. It is not packet-based. It is meant to be consumed character by character.

The reason why you do get chunks is for performance reasons. The operating system groups multiple characters together to a single read/write operation. Note that the decision which characters to group is arbitrarily chosen by the operating system.

In your case, it just looks as if this status info always comes as one packet, because it gets produced so fast that it's smart of the operating system to group it together. Note, however, that there is no guarantee that this is always the case and relying on it is dangerous.

BufReader abstracts all of that away and exposes common functionalities like reading a line of text.

Recommendation:
Use BufReader to process the status information line by line. There is no reliable way to detect the end of the status update, so process each line as a self-contained message. (unless ffmpeg prints some kind of end-of-message marker, like a double newline)

Read portion of lines from child process in Rust (chunk of data)

1 Answers1