java: ProcessBuilder makes a memory hog

Question

I have some issues regarding ProcessBuilder. The program is basically a simple wrapper invoking a command line script.

When running the script on its own via the terminal, the memory consumption stays below 2G. When running the script via the java wrapper, the memory consumption explodes and even 8G is quickly filled up, resulting in out-of-memory errors.

The code to launch the process is simply:

public static int execute(String command) throws IOException
 {
  System.out.println("Executing: " + command);

  ProcessBuilder pb = new ProcessBuilder(command.split(" +"));
  Process p = pb.start();

  // display any output in stderr or stdout  
  StreamConsumer stderr = new StreamConsumer(p.getErrorStream(), "stderr");
  StreamConsumer stdout = new StreamConsumer(p.getInputStream(), "stdout");
  new Thread(stderr).start();
  new Thread(stdout).start();

  try {
   return p.waitFor();
  } catch (InterruptedException e) {
   throw new RuntimeException(e); 
  }
 }

The StreamConsumer class is simply a class which consumes the stdout/stderr streams and display them on the console.

...the question is: why on earth does the memory consumption explode?

Regards,
Arnaud

Edit:

Whether I use ProcessBuilder or Runtime.getRuntime.exec(...), the result is the same.
The memory bursts tend to appear during unix 'sort' invoked by the shell script called:

sort big-text-file > big-text-file.sorted

Edit 2 on request of Jim Garrison:

Ok, here is the StreamConsumer class which I omitted because it is rather simple:

class StreamConsumer implements Runnable
{
    InputStream stream;
    String descr;

    StreamConsumer(InputStream stream, String descr) {
        this.stream = stream;
        this.descr = descr;
    }

    @Override
    public void run()
    {
        String line;

        BufferedReader  brCleanUp = 
            new BufferedReader (new InputStreamReader (stream));

        try {
            while ((line = brCleanUp.readLine ()) != null)
                System.out.println ("[" + descr + "] " + line);
             brCleanUp.close();
        } catch (IOException e) {
            // TODO: handle exception
        }
    }
}

Just a guess: If the StreamConsumer doesn't actually consume the text of the sub-process, the text may be kept in a buffer that constantly increases in size. — Eyal Schneider, Sep 14 '10 at 12:01
All stdout & stderr is successfully displayed in the console — dagnelies, Sep 14 '10 at 12:03
How are you measuring memory consumption? _Which_ process is consuming the memory... the JVM? The shell? The sort process? Without seeing StreamConsumer it's hard to know if there's a bug there. Are you sure memory consumption is private to the process or is it just transient I/O buffers? You really haven't provided much to go on here. — Jim Garrison, Sep 14 '10 at 16:37
We use a queuing system (Sun's grid engine), which measures and limits memory consumption of the jobs. — dagnelies, Sep 15 '10 at 09:09
(this is total memory consumption: the JVM, all child processes and so on.) — dagnelies, Sep 15 '10 at 09:16
Can't you tell which process is cosuming the memory ? It's an important data. — aalku, Jun 21 '11 at 08:22
I don't understand the question. The main process is running out of memory despite it shouldn't ...btw, this issue is 9 month old — dagnelies, Jun 22 '11 at 17:35
This seems to be the same problem as discussed here: http://stackoverflow.com/questions/1124771/how-to-solve-java-io-ioexception-error-12-cannot-allocate-memory-calling-runt — kongo09, Sep 20 '11 at 16:18

score 2 · Answer 1 · answered Jun 23 '11 at 13:54

2

if you change your command like this : sort -o big-text-file.sorted big-text-file

is it always the same ?

answered Jun 23 '11 at 13:54

EricParis16

809
5
9

Good point - perhaps a redirection error. Any reactions to this? – A.H. Oct 15 '11 at 17:31

score 0 · Answer 2 · answered Oct 15 '11 at 16:39

Maybe its because those StreamConsumer threads are not daemons so they don't die and get garbage collected when your processes return? You could try:

//...  
final StreamConsumer stderr = new StreamConsumer(p.getErrorStream(), "stderr");
final StreamConsumer stdout = new StreamConsumer(p.getInputStream(), "stdout");
final Thread stderrThread = new Thread(stderr);
final Thread stdoutThread = new Thread(stdout);
stderrThread.setDaemon(true);
stdoutThread.setDaemon(true);
stderrThread.start();
stdoutThread.start();
//...

Is this behavior happening for single invocation or after doing this many times?

java: ProcessBuilder makes a memory hog

2 Answers2