0

Hello stackoverflow community, I try to run a command on the console and read the input stream of this process:

String command[] = {"ffmpeg"}; //minimal command example
Process proc = processBuilder.command(command).start();
byte[] bytes = IOUtils.toByteArray(proc.getInputStream());
proc.waitFor();

The command is a very long ffmpeg command, which prints bytes to the output. Everything works just fine, but it is very slow using java. When I run this command on my regular command line tool, it takes round about 20ms, everything is printed and done. For the Java process is takes more than 2s. I also tried to redirect all I/O streams to the std, but the performance is the same, since I thought my reading is too slow. Also, reading the stream in additional threads or using other stream readers etc. did not change anything. The only thing which has an effect is adding cmd (currently working with Windows 10) to the command:

String command[] = {"cmd","/C","ffmpeg"};

resulting in a execution time of round about 400ms. I did not know before that this makes a difference, but it really does.

Since this is a Spring Boot web application and the command is used to output images from a video, 400ms is still a lot. The issue here is, that the frontend/browser requests a bunch of images (lets say 36). Apparently the simultaneous requests of a browser to one host is limited (chrome 6 requests), see here. Therefore it takes at best 6 x 400ms to deliver the content. So is there a way to improve the performance to the java process or maybe keep it open and fire commands to avoid overhead?

user2267367
  • 704
  • 7
  • 19
  • If the command produces a lot of output the STDOUT and STDERR must be consumed in the same thread. Otherwise the process freezing or you may get performance issues. – Thilo Schwarz Mar 17 '22 at 11:35
  • Thank you for replying. Ok, but since the process is running in an independent thread by default and is synchronized by process.waitFor() only, how can I consume the output in the same thread? – user2267367 Mar 17 '22 at 12:26
  • 1
    Do you actually need this byte array? – Holger Mar 17 '22 at 17:08
  • Hello Holger, yes at the end I need a byte array. This byte array is basically a image, so a matrix of width x height x color channels. But what would be your idea to perform better? – user2267367 Mar 18 '22 at 04:47
  • 1
    You didn’t put an `@` before my name, so I did not get a notification about your new comment. If you didn’t need the array, you could have used a native redirection to avoid reading it. In fact, even as you need the data, a native redirection could turn out to be faster than the pipe. Try, e.g. `File f = File.createTempFile("ffmpeg", null); new ProcessBuilder("ffmpeg").redirectOutput(f) .start().waitFor(); byte[] array = Files.readAllBytes(f.toPath()); f.delete();` When you’re using Java 9 or newer, you could also try `proc.getInputStream().readAllBytes()`. – Holger Mar 21 '22 at 18:29
  • Hi @Holger, sorry for the missing @. I tried your solution with the file. I tried to avoid the HDD/SSD, since it is relatively slow compared to the RAM. But your way with the file is just as fast as reading the stream, not really better, not worse. So I guess this part isn't the bottleneck – user2267367 Mar 23 '22 at 09:33
  • 1
    How large is the output (in terms of bytes)? And how does you fast setup look like (when it takes only 20ms)? Printing to the console or piping to another tool? – Holger Mar 23 '22 at 16:33

1 Answers1

0

Here is the pseudo code to your question in the comments above:

        ByteArrayOutputStream out = new ByteArrayOutputStream();
        ProcessBuilder pb = new ProcessBuilder(command);
            
        // Consuming STDOUT and STDERR in same thread can lead to the process freezing if it writes large amounts.
        pb.redirectErrorStream(true);
    
        Process process = pb.start();
        try (var infoStream = process.getInputStream()) {
            infoStream.transferTo(out);
        }
        status = process.waitFor();
        if (status > 0) 
            // errorhandling

#transferTo(out) Consumes the output stream inside the processbuilder thread. That's it! It runs inside the thread of process.

Thilo Schwarz
  • 640
  • 5
  • 24