java.io.IOException: error=11

Question

I am experiencing a weird problem with the Java ProcessBuilder. The code is shown below (in a slightly simplified form)

public class Whatever implements Runnable
{

public void run(){
        //someIdentifier is a randomly generated string
        String in = someIdentifier + "input.txt";
        String out = someIdentifier + "output.txt";
        ProcessBuilder builder = new ProcessBuilder("./whateveer.sh", in, out);
        try {
            Process process = builder.start();
            process.waitFor();
        } catch (IOException e) {
            log.error("Could not launch process. Command: " + builder.command(), e);
        } catch (InterruptedException ex) {
            log.error(ex);
        }
}

}

whatever.sh reads:

R --slave --args $1 $2 <whatever1.R >> r.log

Loads of instances of Whatever are submitted to an ExecutorService of fixed size (35). The rest of the application waits for all of them to finish- implemented with a CountdownLatch. Everything runs fine for several hours (Scientific Linux 5.0, java version "1.6.0_24") before throwing the following exception:

java.io.IOException: Cannot run program "./whatever.sh": java.io.IOException: error=11, Resource temporarily unavailable
    at java.lang.ProcessBuilder.start(Unknown Source)
... rest of stack trace omitted...

Does anyone have an idea what this means? Based on the google/bing search results for java.io.IOException: error=11, it is not the most common of exceptions and I am completely baffled.

My wild and not so educated guess is that I have too many threads trying to launch the same file at the same time. However, it takes hours of CPU time to reproduce the problem, so I have not tried with a smaller number.

Any suggestions are greatly appreciated.

have you checked the open file cound of your java process with `lsof`? — lidaobing, Dec 05 '11 at 10:38

score 9 · Accepted Answer · answered Dec 05 '11 at 10:54

The error=11 is almost certainly the EAGAIN error code:

$ grep EAGAIN asm-generic/errno-base.h 
#define EAGAIN      11  /* Try again */

The clone(2) system call documents an EAGAIN error return:

   EAGAIN Too many processes are already running.

The fork(2) system call documents two EAGAIN error returns:

   EAGAIN fork() cannot allocate sufficient memory to copy the
          parent's page tables and allocate a task structure for
          the child.

   EAGAIN It was not possible to create a new process because
          the caller's RLIMIT_NPROC resource limit was
          encountered.  To exceed this limit, the process must
          have either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE
          capability.

If you were really that low on memory, it would almost certainly show in the system logs. Check dmesg(1) output or /var/log/syslog for any potential messages about low system memory. (Other things would break. This doesn't seem too plausible.)

Much more likely is running into either the per-user limit on processes or system-wide maximum number of processes. Perhaps one of your processes isn't properly reapting zombies? This would be very easy to spot by checking ps(1) output over time:

while true ; do ps auxw >> ~/processes ; sleep 10 ; done

(Maybe check every minute or ten minutes if it really does take hours before you're in trouble.)

If you're not reaping zombies, then read up on whatever you must do to ProcessBuilder to use waitpid(2) to reap your dead children.

If you're legitimately running more processes than your rlimits allow, you'll need to use ulimit in your bash(1) scripts (if running as root) or set higher limits in /etc/security/limits.conf for the nproc property.

If you are instead running into the system-wide process limits, you might need to write a larger value into /proc/sys/kernel/pid_max. See proc(5) for some (short) details.

Thanks, that lead me in the right direction. Due to a bug in the code a new thread was getting created in a loop instead of re-using an existing one, eventually resulting in the problem described. — mbatchkarov, Dec 06 '11 at 09:11
So it wasn't the `builder.start()` call _specifically_ that was at fault, but just the piece of code that wound up reporting the problem? Good debugging. :) — sarnold, Dec 06 '11 at 09:17
Well, it really was `builder.start()` that failed, but it failed because another part of the application had already exhausted the available resources. As you mentioned in your post, forking a process requires a certain amount of memory etc. Thanks for your help — mbatchkarov, Dec 06 '11 at 09:22

score 2 · Answer 2 · answered Dec 05 '11 at 10:46

errno 11 means "Resource temporarily unavailable" This is usually a memory problem and can prevent a thread or socket being created.

errno 12 means "Can't allocate memory". This is a failure to obtain memory is a direct call for memory (rather than a resource which in turn needs memory)

I would try increasing the swap space of your system which should avoid this issue.

java.io.IOException: error=11

2 Answers2

Linked