2

I'm programming a little GUI for a file converter in java. The file converter writes its current progress to stdout. Looks like this:

Flow_1.wav: 28% complete, ratio=0,447

I wanted to illustrate this in a progress bar, so I'm reading the process' stdout like this:

ProcessBuilder builder = new ProcessBuilder("...");
builder.redirectErrorStream(true);
Process proc = builder.start();
InputStream stream = proc.getInputStream();
byte[] b = new byte[32];
int length;
while (true) {
    length = stream.read(b);
    if (length < 0) break;
    // processing data
}

Now the problem is that regardless which byte array size I choose, the stream is read in chunks of 4 KB. So my code is being executed until length = stream.read(b); and then blocks for quite a while. Once the process generates 4 KB output data, my programm gets this chunk and works through it in 32 byte slices. And then waits again for the next 4 KB.

I tried to force java to use smaller buffers like this:

BufferedInputStream stream = new BufferedInputStream(proc.getInputStream(), 32);

Or this:

BufferedReader reader = new BufferedReader(new InputStreamReader(proc.getInputStream()), 32);

But neither changed anything.

Then I found this: Process source (around line 87)

It seems that the Process class is implemented in such a way that it pipes the process' stdout to a file. So what proc.getInputStream(); actually does, is returning a stream to a file. And this file seems to be written with a 4 KB buffer.

Does anyone know some kind of workaround for this situation? I just want to get the process' output instantly.

EDIT: As suggested by Ian Roberts, I also tried to pipe the converter's output into the stderr stream, since this stream doesn't seem to be wrapped in a BufferedInputStream. Still 4k chunks.

Another interesting thing is: I actually don't get exactly 4096 bytes, but about 5 more. I'm afraid the FileInputStream itself is buffered natively.

R2-D2
  • 1,554
  • 1
  • 13
  • 25
  • 2
    not that it changes your issue, but java _isn't_ streaming the data to a file. file descriptors are how operating systems describe the process input/output streams. they don't necessarily correlate to _real_ files (although they might). – jtahlborn Feb 19 '13 at 17:36
  • Usually stdout is already buffered by the process that writes it, so if this is case, you have no chance if you cannot change the other process. That process would need to turn off buffering or call flush() more often. – Philipp Wendler Feb 20 '13 at 11:23
  • @PhilippWendler: The converter's stdout definitively gets flushed more often than every 4 kb (when run from terminal). Do you think the automatic flush at \n gets turned off when stdout is not being sent to a console? – R2-D2 Feb 20 '13 at 11:41
  • @R2-D2 While it's technically possible, I wouldn't expect an application to do so. I know that the OS also might do buffering, but I don't know any details about that. Perhaps investigate into that direction? – Philipp Wendler Feb 20 '13 at 16:22
  • You could write a small C program that starts your process and reads the input to see whether it's a Java-specific problem or not. – Philipp Wendler Feb 20 '13 at 16:23
  • I can't find how to capture other processes' stdout streams in c. But I found a ruby solution using pseudo terminals. ([link](http://stackoverflow.com/questions/1154846/continuously-read-from-stdout-of-external-process-in-ruby?rq=1)) Unfortunately there's no java equivalent (afaik). – R2-D2 Feb 20 '13 at 23:38
  • Aw, I'm stupid. The converter **has to be flushing his stdout** regulary, because the progress is printed to the same line all the time. It uses \r, _not_ \n. \r doesn't auto-flush, so it should have to call `fflush`. This makes no sense to me! – R2-D2 Feb 20 '13 at 23:54

1 Answers1

2

Looking at the code you linked to the process's standard output stream gets wrapped in a BufferedInputStream but its standard error remains unbuffered. So one possibility might be to execute not the converter directly, but a shell script (or Windows equivalent if you're on Windows) that sends the converter's stdout to stderr:

ProcessBuilder builder = new ProcessBuilder("/bin/sh", "-c",
  "exec /path/to/converter args 1>&2");

Don't redirectErrorStream, and then read from proc.getErrorStream() instead of proc.getInputStream().

It may be the case that your converter is already using stderr for its progress reporting in which case you don't need the script bit, just turn off redirectErrorStream(). If the converter program writes to both stdout and stderr then you'll need to spawn a second thread to consume stdout as well (the script approach gets around this by sending everything to stderr).

Ian Roberts
  • 120,891
  • 16
  • 170
  • 183