In Windows: Get unbuffered stdout from C program into Java program InputStream [Now with minimal example!]

Question

SOLVED:

HovecraftFullOfEels found my mistake. See the comments on his answer below. I was calling run() on a Thread object when I should have been calling start(). In my original program, that meant it blocked on stderr until the program quit and then got stdout all at once.

UPDATE:

As requested, I've produced a minimal example. Here's the producer:

package producer;

public class Producer {

    public static void main(String[] args) throws InterruptedException {
        for (int i=0; i<10; i++) {
            System.out.println(i);
            System.out.flush();
            Thread.sleep(1000);
        }
    }    
}

Here's the consumer:

package consumer;

import java.io.IOException;
import java.io.InputStream;

public class Consumer {
    static class Dumper extends Thread {
        InputStream r;
        boolean print;

        Dumper(InputStream r, boolean print) {
            this.r = r;
            this.print = print;
        }

        @Override
        public void run() {
            int c;
            try {
                while ((c = r.read()) != -1) {
                    if (print) System.out.print((char)c);
                }
                r.close();
            } catch (IOException ex) {}
        }
    }

    public static void main(String[] args) throws Exception {
        Process p = Runtime.getRuntime().exec("java -jar C:\\Users\\millerti\\Documents\\NetBeansProjects\\Producer\\dist\\Producer.jar");
        new Dumper(p.getErrorStream(), false).run();
        new Dumper(p.getInputStream(), true).run();
        p.waitFor();
    }
}

Expected behavior: Since the producer flushes its output and the consumer is completely unbuffered, the consumer should get the producer's output in real time.

Observed behavior: The consumer receives the producer's output all at once only when the producer finished.

Original post:

I'm working with another developer on a Windows application. She has written a C program (which we can modify) that performs a media conversion and prints progress messages (in percentage) to stdout. I am writing a graphical front-end in Java, and I would like my program to capture those messages and update a GUI element.

The relevant C program code looks like this:

printf ("progress_msg: %d%%\n", currentProgress);
fflush(stdout);

In the Java program, I'm using Runtime.getRuntime().exec() to run the C program and grabbing the raw InputStream objects for C program's stdout and stderr. (The stderr is just being thrown away by another thread.) I'm parsing the stdout stream, looking for those messages (well, I was, using a BufferedReader, but right now, I'm trying to debug this, so I'm talking directly to the low-level InputStream byte source.)

The problem is that although there is no buffering on the Java side (that I know of) and there's an fflush in the C program, the stdout text appears all at once only when the C program finishes. This is unlike when I run it from the command prompt, where the messages come out gradually as expected.

So clearly, there's some buffering going on. Is the Process facility in Java doing some buffering that I don't know about? If so, is there a way to turn it off? Does fflush(stdout) in Windows not work in the expected way? What is the proper Windows equivalent?

Thanks.

Note: I have found two related stackoverflows, but they do not answer this question. In particular, we CAN modify the C program, there's no line buffering, and we ARE doing a proper flush (as far as we know). See I want realtime output of my Runtime.getRuntime().exec() and Unbuffered subprocess stdout on windows.

Answer 1

A wild guess from my part, but you state:

... I am writing a graphical front-end in Java, and I would like my program to capture those messages and update a GUI element.
...

In the Java program, I'm using Runtime.getRuntime().exec() to run the C program and grabbing the raw InputStream objects for C program's stdout and stderr. ... I'm parsing the stdout stream, looking for those messages ...

The problem is that although there is no buffering on the Java side (that I know of) and there's an fflush in the C program, the stdout text appears all at once only when the C program finishes.

I again have to guess as you've left out important details, all code, but if your front end is a Swing GUI, then the symptoms you've described suggest that you're trying to get all information on the Swing event thread. If this is a Swing application, a solution is to make sure to read the output of the Stream in a background thread such as that provided by a SwingWorker, to make sure that you buffer this data that you're reading in, and to then display it in the GUI in a thread-safe way (again using a SwingWorker, especially its publish/process method pair).

For better help, please give us better information and code, preferably a small, in fact minimal, example program that we can run and test and improve.

---

Edit
My test program:

import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;

public class Consumer {

   static class StreamGobbler implements Runnable {
      private String name;
      private boolean print;
      private Scanner scanner;

      public StreamGobbler(String name, InputStream inputStream, boolean print) {
         this.name = name;
         this.print = print;

         scanner = new Scanner(inputStream);
      }

      @Override
      public void run() {
         System.out.printf("in %s run method%n", name);
         while (scanner.hasNextLine()) {
            String line = scanner.nextLine();
            if (print) {
               String output = String.format("From %s: %s%n", name, line);
               System.out.printf(output);
            }
         }
         if (scanner != null) {
            scanner.close();
         }
      }
   }

   private static final String PRODUCER_PATH = "C:/Users/Pete/Documents/Fubar/Java/producer.jar";

   public static void main(String[] args) {
      List<String> command = new ArrayList<>();
      command.add("java.exe");
      command.add("-jar");
      command.add(PRODUCER_PATH);

      try {
         ProcessBuilder pBuilder = new ProcessBuilder(command);
         Process process = pBuilder.start(); 
         StreamGobbler errorGobbler = new StreamGobbler("Error Gobbler", process.getErrorStream(), true);
         StreamGobbler inputGobbler = new StreamGobbler("Input Gobbler", process.getInputStream(), true);

         new Thread(errorGobbler).start();
         new Thread(inputGobbler).start();
         process.waitFor();
      } catch (IOException | InterruptedException e) {
         e.printStackTrace();
      }
   }
}