ProcessBuilder: Forwarding stdout and stderr of started processes without blocking the main thread

Question

I'm building a process in Java using ProcessBuilder as follows:

ProcessBuilder pb = new ProcessBuilder()
        .command("somecommand", "arg1", "arg2")
        .redirectErrorStream(true);
Process p = pb.start();

InputStream stdOut = p.getInputStream();

Now my problem is the following: I would like to capture whatever is going through stdout and/or stderr of that process and redirect it to System.out asynchronously. I want the process and its output redirection to run in the background. So far, the only way I've found to do this is to manually spawn a new thread that will continuously read from stdOut and then call the appropriate write() method of System.out.

new Thread(new Runnable(){
    public void run(){
        byte[] buffer = new byte[8192];
        int len = -1;
        while((len = stdOut.read(buffer)) > 0){
            System.out.write(buffer, 0, len);
        }
    }
}).start();

While that approach kind of works, it feels a bit dirty. And on top of that, it gives me one more thread to manage and terminate correctly. Is there any better way to do this?

If blocking the calling thread was an option, there'd be a very simple solution even in Java 6: `org.apache.commons.io.IOUtils.copy(new ProcessBuilder().command(commandLine) .redirectErrorStream(true).start().getInputStream(), System.out);` — oberlies, Aug 09 '13 at 12:29

score 178 · Answer 1 · edited Mar 16 '16 at 11:31

178

Use ProcessBuilder.inheritIO, it sets the source and destination for subprocess standard I/O to be the same as those of the current Java process.

Process p = new ProcessBuilder().inheritIO().command("command1").start();

If Java 7 is not an option

public static void main(String[] args) throws Exception {
    Process p = Runtime.getRuntime().exec("cmd /c dir");
    inheritIO(p.getInputStream(), System.out);
    inheritIO(p.getErrorStream(), System.err);

}

private static void inheritIO(final InputStream src, final PrintStream dest) {
    new Thread(new Runnable() {
        public void run() {
            Scanner sc = new Scanner(src);
            while (sc.hasNextLine()) {
                dest.println(sc.nextLine());
            }
        }
    }).start();
}

Threads will die automatically when subprocess finishes, because src will EOF.

edited Mar 16 '16 at 11:31

Chirlo

5,989
1
29
45

answered Jan 05 '13 at 02:54

Evgeniy Dorofeev

133,369
30
199
275

1

I see that Java 7 has added a bunch of interesting methods for handling stdout, stderr and stdin. Pretty nice. I think I'll use `inheritIO()` or one of those handy `redirect*(ProcessBuilder.Redirect)` methods next time I need to do that on a java 7 project. Unfortunately my project is java 6. – LordOfThePigs Jan 05 '13 at 05:38
1

`sc` needs to be closed? – hotohoto Jan 17 '17 at 07:03
Note that it sets it to the OS filedescriptor of the parent JVM, not to the System.out streams. This is therefore ok to write to console or shell redirection of the parent but it wont work for logging streams. Those still need a pump thread (however you can at least redirect stderr to stdin, so you only need one thread. – eckes Jul 26 '18 at 18:50

score 74 · Accepted Answer · edited Apr 11 '21 at 01:39

74

For Java 7 and later, see Evgeniy Dorofeev's answer.

For Java 6 and earlier, create and use a StreamGobbler:

StreamGobbler errorGobbler = 
  new StreamGobbler(p.getErrorStream(), "ERROR");

// any output?
StreamGobbler outputGobbler = 
  new StreamGobbler(p.getInputStream(), "OUTPUT");

// start gobblers
outputGobbler.start();
errorGobbler.start();

...

private class StreamGobbler extends Thread {
    InputStream is;
    String type;

    private StreamGobbler(InputStream is, String type) {
        this.is = is;
        this.type = type;
    }

    @Override
    public void run() {
        try {
            InputStreamReader isr = new InputStreamReader(is);
            BufferedReader br = new BufferedReader(isr);
            String line = null;
            while ((line = br.readLine()) != null)
                System.out.println(type + "> " + line);
        }
        catch (IOException ioe) {
            ioe.printStackTrace();
        }
    }
}

edited Apr 11 '21 at 01:39

Dave Jarvis

30,436
41
178
315

answered Jan 04 '13 at 21:50

asgoth

35,552
12
89
98

1

Will the StreamGobbler catch all of the output? Is there any chance of it missing some of the output? Also will the StreamGobbler thread die on its own once the process is stopper? – LordOfThePigs Jan 04 '13 at 22:01
2

From the moment, the InputStream has ended, it will end too. – asgoth Jan 04 '13 at 22:03
7

For Java 6 and earlier, it seems this is the only solution. For java 7 and up, see the other answer about ProcessBuilder.inheritIO() – LordOfThePigs Jan 05 '13 at 05:39
@asgoth Is there any way to send input to the process? Here is my question : https://stackoverflow.com/questions/28070841/javafx-redirect-input-output-error-from-processbuilder-to-textarea, I will be grateful if someone will help me to solve the problem. – DeepSidhu1313 Jan 27 '15 at 18:29

Adam Michalik · Answer 3 · 2016-12-22T15:26:35.150

A flexible solution with Java 8 lambda that lets you provide a Consumer that will process the output (eg. log it) line by line. run() is a one-liner with no checked exceptions thrown. Alternatively to implementing Runnable, it can extend Thread instead as other answers suggest.

class StreamGobbler implements Runnable {
    private InputStream inputStream;
    private Consumer<String> consumeInputLine;

    public StreamGobbler(InputStream inputStream, Consumer<String> consumeInputLine) {
        this.inputStream = inputStream;
        this.consumeInputLine = consumeInputLine;
    }

    public void run() {
        new BufferedReader(new InputStreamReader(inputStream)).lines().forEach(consumeInputLine);
    }
}

You can then use it for example like this:

public void runProcessWithGobblers() throws IOException, InterruptedException {
    Process p = new ProcessBuilder("...").start();
    Logger logger = LoggerFactory.getLogger(getClass());

    StreamGobbler outputGobbler = new StreamGobbler(p.getInputStream(), System.out::println);
    StreamGobbler errorGobbler = new StreamGobbler(p.getErrorStream(), logger::error);

    new Thread(outputGobbler).start();
    new Thread(errorGobbler).start();
    p.waitFor();
}

Here the output stream is redirected to System.out and the error stream is logged on the error level by the logger.

@Robert The threads will stop automatically when the corresponding input/error stream is closed. The `forEach()` in the `run()` method will block until the stream is open, awaiting the next line. It will exit when the stream is closed. — Adam Michalik, Jun 08 '19 at 17:44
Expanding on this, if you want to return the data from the process you should always join the thread. It could happen that `p.waitFor()` finishes and the buffer is still being read. — digijap, Dec 02 '21 at 13:23

nike.laos · Answer 4 · 2017-06-29T09:01:54.067

It's as simple as following:

    File logFile = new File(...);
    ProcessBuilder pb = new ProcessBuilder()
        .command("somecommand", "arg1", "arg2")
    processBuilder.redirectErrorStream(true);
    processBuilder.redirectOutput(logFile);

by .redirectErrorStream(true) you tell process to merge error and output stream and then by .redirectOutput(file) you redirect merged output to a file.

Update:

I did manage to do this as follows:

public static void main(String[] args) {
    // Async part
    Runnable r = () -> {
        ProcessBuilder pb = new ProcessBuilder().command("...");
        // Merge System.err and System.out
        pb.redirectErrorStream(true);
        // Inherit System.out as redirect output stream
        pb.redirectOutput(ProcessBuilder.Redirect.INHERIT);
        try {
            pb.start();
        } catch (IOException e) {
            e.printStackTrace();
        }
    };
    new Thread(r, "asyncOut").start();
    // here goes your main part
}

Now you're able to see both outputs from main and asyncOut threads in System.out

This does not answer the question: *I would like to capture whatever is going through stdout and/or stderr of that process and redirect it to System.out asynchronously. I want the process and its output redirection to run in the background.* — Adam Michalik, Jun 28 '17 at 19:12
@AdamMichalik, you're right - I didn't get the gist, at first. Thanks for showing. — nike.laos, Jun 29 '17 at 09:05
This has the same problem as inheritIO(), it will write to the poarent JVMs FD1 but not to any replaces System.out OutputStreams (like logger adapter). — eckes, Jul 26 '18 at 18:52

score 4 · Answer 5 · edited Mar 29 '21 at 17:08

Simple java8 solution with capturing both outputs and reactive processing using CompletableFuture:

static CompletableFuture<String> readOutStream(InputStream is) {
    return CompletableFuture.supplyAsync(() -> {
        try (
                InputStreamReader isr = new InputStreamReader(is);
                BufferedReader br = new BufferedReader(isr);
        ) {
              StringBuilder res = new StringBuilder();
              String inputLine;
              while ((inputLine = br.readLine()) != null) {
                  res.append(inputLine).append(System.lineSeparator());
              }
              return res.toString();
        } catch (Throwable e) {
            throw new RuntimeException("problem with executing program", e);
        }
    });
}

And the usage:

Process p = Runtime.getRuntime().exec(cmd);
CompletableFuture<String> soutFut = readOutStream(p.getInputStream());
CompletableFuture<String> serrFut = readOutStream(p.getErrorStream());
CompletableFuture<String> resultFut = 
    soutFut.thenCombine(serrFut, (stdout, stderr) -> {
         // print to current stderr the stderr of process and return the stdout
         System.err.println(stderr);
        
         return stdout;
    });
// get stdout once ready, blocking
String result = resultFut.get();

This solution is very straight forward. It also shows indirectly how to redirect i.e. to a logger. For an example have a look at my answer. — keocra, Mar 06 '20 at 10:11
This looks clean, CompletableFutures will use additional threads from the default thread executor, which is not as efficient as could be watching file descriptors for stout and stderr for changes and then reacting. Yet as for version 18, this is so far the best option that we can do in Java. — Luke 10X, Apr 07 '23 at 17:37

mryan · Answer 6 · 2020-04-20T09:15:52.790

There is a library that provides a better ProcessBuilder, zt-exec. This library can do exactly what you are asking for and more.

Here's what your code would look like with zt-exec instead of ProcessBuilder :

add the dependency :

<dependency>
  <groupId>org.zeroturnaround</groupId>
  <artifactId>zt-exec</artifactId>
  <version>1.11</version>
</dependency>

The code :

new ProcessExecutor()
  .command("somecommand", "arg1", "arg2")
  .redirectOutput(System.out)
  .redirectError(System.err)
  .execute();

Documentation of the library is here : https://github.com/zeroturnaround/zt-exec/

score 3 · Answer 7 · answered Feb 07 '14 at 19:31

I too can use only Java 6. I used @EvgeniyDorofeev's thread scanner implementation. In my code, after a process finishes, I have to immediately execute two other processes that each compare the redirected output (a diff-based unit test to ensure stdout and stderr are the same as the blessed ones).

The scanner threads don't finish soon enough, even if I waitFor() the process to complete. To make the code work correctly, I have to make sure the threads are joined after the process finishes.

public static int runRedirect (String[] args, String stdout_redirect_to, String stderr_redirect_to) throws IOException, InterruptedException {
    ProcessBuilder b = new ProcessBuilder().command(args);
    Process p = b.start();
    Thread ot = null;
    PrintStream out = null;
    if (stdout_redirect_to != null) {
        out = new PrintStream(new BufferedOutputStream(new FileOutputStream(stdout_redirect_to)));
        ot = inheritIO(p.getInputStream(), out);
        ot.start();
    }
    Thread et = null;
    PrintStream err = null;
    if (stderr_redirect_to != null) {
        err = new PrintStream(new BufferedOutputStream(new FileOutputStream(stderr_redirect_to)));
        et = inheritIO(p.getErrorStream(), err);
        et.start();
    }
    p.waitFor();    // ensure the process finishes before proceeding
    if (ot != null)
        ot.join();  // ensure the thread finishes before proceeding
    if (et != null)
        et.join();  // ensure the thread finishes before proceeding
    int rc = p.exitValue();
    return rc;
}

private static Thread inheritIO (final InputStream src, final PrintStream dest) {
    return new Thread(new Runnable() {
        public void run() {
            Scanner sc = new Scanner(src);
            while (sc.hasNextLine())
                dest.println(sc.nextLine());
            dest.flush();
        }
    });
}

score 2 · Answer 8 · answered Jan 10 '21 at 02:44

It's really surprising to me that the redirection methods in ProcessBuilder don't accept an OutputStream, only File. Yet another proof of forced boilerplate code that Java forces you to write.

That said, let's look at a list of comprehensive options:

If you want the process output to simply be redirected to its parent's output stream, inheritIO will do the job.
If you want the process output to go to a file, use redirect*(file).
If you want the process output to go to a logger, you need to consume the process InputStream in a separate thread. See the answers that use a Runnable or CompletableFuture. You can also adapt the code below to do this.
If you want to the process output to go to a PrintWriter, that may or may not be the stdout (very useful for testing), you can do the following:

static int execute(List<String> args, PrintWriter out) {
    ProcessBuilder builder = new ProcessBuilder()
            .command(args)
            .redirectErrorStream(true);
    Process process = null;
    boolean complete = false;
    try {
        process = builder.start();
        redirectOut(process.getInputStream(), out)
                .orTimeout(TIMEOUT, TimeUnit.SECONDS);
        complete = process.waitFor(TIMEOUT, TimeUnit.SECONDS);
    } catch (IOException e) {
        throw new UncheckedIOException(e);
    } catch (InterruptedException e) {
        LOG.warn("Thread was interrupted", e);
    } finally {
        if (process != null && !complete) {
            LOG.warn("Process {} didn't finish within {} seconds", args.get(0), TIMEOUT);
            process = process.destroyForcibly();
        }
    }

    return process != null ? process.exitValue() : 1;
}

private static CompletableFuture<Void> redirectOut(InputStream in, PrintWriter out) {
    return CompletableFuture.runAsync(() -> {
        try (
                InputStreamReader inputStreamReader = new InputStreamReader(in);
                BufferedReader bufferedReader = new BufferedReader(inputStreamReader)
        ) {
            bufferedReader.lines()
                    .forEach(out::println);
        } catch (IOException e) {
            LOG.error("Failed to redirect process output", e);
        }
    });
}

Advantages of the code above over the other answers thus far:

redirectErrorStream(true) redirects the error stream to the output stream, so that we only have to bother with one.
CompletableFuture.runAsync runs from the ForkJoinPool. Note that this code doesn't block by calling get or join on the CompletableFuture but sets a timeout instead on its completion (Java 9+). There's no need for CompletableFuture.supplyAsync because there's nothing really to return from the method redirectOut.
BufferedReader.lines is simpler than using a while loop.

score 1 · Answer 9 · answered Mar 06 '20 at 10:21

As an addition to msangel answer I would like to add the following code block:

private static CompletableFuture<Boolean> redirectToLogger(final InputStream inputStream, final Consumer<String> logLineConsumer) {
        return CompletableFuture.supplyAsync(() -> {
            try (
                InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
                BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
            ) {
                String line = null;
                while((line = bufferedReader.readLine()) != null) {
                    logLineConsumer.accept(line);
                }
                return true;
            } catch (IOException e) {
                return false;
            }
        });
    }

It allows to redirect the input stream (stdout, stderr) of the process to some other consumer. This might be System.out::println or anything else consuming strings.

Usage:

...
Process process = processBuilder.start()
CompletableFuture<Boolean> stdOutRes = redirectToLogger(process.getInputStream(), System.out::println);
CompletableFuture<Boolean> stdErrRes = redirectToLogger(process.getErrorStream(), System.out::println);
System.out.println(stdOutRes.get());
System.out.println(stdErrRes.get());
System.out.println(process.waitFor());

score 0 · Answer 10 · answered Mar 29 '18 at 07:05

Thread thread = new Thread(() -> {
      new BufferedReader(
          new InputStreamReader(inputStream, 
                                StandardCharsets.UTF_8))
              .lines().forEach(...);
    });
    thread.start();

Your custom code goes instead of the ...

score -1 · Answer 11 · answered Oct 26 '20 at 18:46

import java.io.BufferedReader;
import java.io.InputStreamReader;

public class Main {

    public static void main(String[] args) throws Exception {
        ProcessBuilder pb = new ProcessBuilder("script.bat");
        pb.redirectErrorStream(true);
        Process p = pb.start();
        BufferedReader logReader = new BufferedReader(new InputStreamReader(p.getInputStream()));
        String logLine = null;
        while ( (logLine = logReader.readLine()) != null) {
           System.out.println("Script output: " + logLine);
        }
    }
}

By using this line: pb.redirectErrorStream(true); we can combine InputStream and ErrorStream

That's litterally already used in the code that's in the question... — LordOfThePigs, Oct 27 '20 at 08:50

score -2 · Answer 12 · answered Oct 01 '15 at 17:21

By default, the created subprocess does not have its own terminal or console. All its standard I/O (i.e. stdin, stdout, stderr) operations will be redirected to the parent process, where they can be accessed via the streams obtained using the methods getOutputStream(), getInputStream(), and getErrorStream(). The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, or even deadlock.

https://www.securecoding.cert.org/confluence/display/java/FIO07-J.+Do+not+let+external+processes+block+on+IO+buffers

ProcessBuilder: Forwarding stdout and stderr of started processes without blocking the main thread

12 Answers12

Linked

Related