Java: is there a way to run a system command and print the output during execution?

Question

I have a python script and it takes a long time to finish. I would like to run it from Java, but also output the script's output while it is executing, so that I can tell if it is properly running.

I've searched and only found examples where we output the output after the system command has finished, rather than during its execution.

Any way to do it while the script is running?

Here's what I have

public void doSomething() throws IOException {
    String[] callAndArgs = {"python", "/hi.py"};
    Process p = Runtime.getRuntime().exec(callAndArgs);
    BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
    BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));

    String s;
    while ((s = stdInput.readLine()) != null) {
        System.out.println(s);
    }

    while ((s = stdError.readLine()) != null) {
        System.out.println(s);
    }
}

Answer 1

i managed to get it working like this (Note it requires java7):

package test;
import java.lang.ProcessBuilder.Redirect;

public class Test {

    public static void main(String... args) throws Exception {
        ProcessBuilder pb = new ProcessBuilder("python","/home/foobar/Programming/test/src/test/test.py");
        pb.redirectOutput(Redirect.INHERIT);
        Process p = pb.start();
        p.waitFor();
    }

}

python (note i flush on python to make it work using sys.stdout.flush())

import time,sys
c =0
while c<=50:
    time.sleep(1)
    print("----")
    c = c +1
    sys.stdout.flush()

Note if you don't want to flush in a loop you can use this:

ProcessBuilder pb = new ProcessBuilder("python","-u","/home/foobar/Programming/NetBeansProjects/test/src/test/test.py");

Redirect.INHERIT

Indicates that subprocess I/O source or destination will be the same as those of the current process. This is the normal behavior of most operating system command interpreters (shells).

Answer 2

I've searched and only found examples where we output the output after the system command has finished, rather than during its execution.

That's weird, because your example should be dumping the output as the command is executing.

Instead of using BufferedReader, you could try reading directly from the InputStream instead as the required conditions for readLine might not be being met until after the process exits.

I'd also recommend that you use a ProcessBuilder over Process directly, as, apart from anything else, it allows you to redirect the output from the error stream into the input stream, allowing you to read just one stream instead of two...

This might also be an issue with Python and how it flushes it output buffers...

For example, rather then waiting for the BufferedReader to decide when to return, try printing each character from the stream as it occurs/is reported

            ProcessBuilder pb = new ProcessBuilder("test.py");
            pb.redirectError();
            Process p = pb.start();

            InputStream is = null;
            try {
                is = p.getInputStream();
                int in = -1;
                while ((in = is.read()) != -1) {
                    System.out.print((char)in);
                }
            } finally {
                try {
                    is.close();
                } catch (Exception e) {
                }
            }

Update

Doing a little reading, Python seems to be buffering its out out before sending it to the stdout. I don't think you can fix this on the a Java side, but need to alter either the way Python is run or the script works.

See How to flush output of Python print? for more details

Answer 3

I'm suspecting that you are writing to stderr, which you can't see because you are blocking on stdin. Use a ProcessBuilder instead of doing exec. This way, you can redirect stderr and stdin into a single stream.

Here is an example:

import java.io.*;

public class Test {
    public static void main(String... args) throws IOException {

        ProcessBuilder pb =
                new ProcessBuilder("test.py");

        pb.redirectErrorStream(true);
        Process proc = pb.start();

        Reader reader = new InputStreamReader(proc.getInputStream());
        BufferedReader bf = new BufferedReader(reader);
        String s;
        while ((s = bf.readLine()) != null) {
            System.out.println(s);
        }
    }
}

Alternatively you can spawn threads to read from stdin/stderr respectively.

Another thing to look for is output buffering by python. You can see if this is the cause by doing:

import sys
sys.stdout.flush()

after you write to stdout

Answer 4

Don't use #readLine as the conditional in your while loop. Instead wrap your inputStream in a scanner and use #hasNextLine()

    Scanner in = new Scanner(p.getInputStream());
    while (in.hasNextLine()) {
        System.out.println(in.nextLine());
    }