29

I'm trying to debug a file descriptor leak in a Java webapp running in Jetty 7.0.1 on Linux.

The app had been happily running for a month or so when requests started to fail due to too many open files, and Jetty had to be restarted.

java.io.IOException: Cannot run program [external program]: java.io.IOException: error=24, Too many open files
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
    at java.lang.Runtime.exec(Runtime.java:593)
    at org.apache.commons.exec.launcher.Java13CommandLauncher.exec(Java13CommandLauncher.java:58)
    at org.apache.commons.exec.DefaultExecutor.launch(DefaultExecutor.java:246)

At first I thought the issue was with the code that launches the external program, but it's using commons-exec and I don't see anything wrong with it:

CommandLine command = new CommandLine("/path/to/command")
    .addArgument("...");
ByteArrayOutputStream errorBuffer = new ByteArrayOutputStream();
Executor executor = new DefaultExecutor();
executor.setWatchdog(new ExecuteWatchdog(PROCESS_TIMEOUT));
executor.setStreamHandler(new PumpStreamHandler(null, errorBuffer));
try {
    executor.execute(command);
} catch (ExecuteException executeException) {
    if (executeException.getExitValue() == EXIT_CODE_TIMEOUT) {
        throw new MyCommandException("timeout");
    } else {
        throw new MyCommandException(errorBuffer.toString("UTF-8"));
    }
}

Listing open files on the server I can see a high number of FIFOs:

# lsof -u jetty
...
java    524 jetty  218w  FIFO        0,6      0t0 19404236 pipe
java    524 jetty  219r  FIFO        0,6      0t0 19404008 pipe
java    524 jetty  220r  FIFO        0,6      0t0 19404237 pipe
java    524 jetty  222r  FIFO        0,6      0t0 19404238 pipe

when Jetty starts there are just 10 FIFOs, after a few days there are hundreds of them.

I know it's a bit vague at this stage, but do you have any suggestions on where to look next, or how to get more detailed info about those file descriptors?

Mirko N.
  • 10,537
  • 6
  • 38
  • 37
  • Added the code launching the external program. – Mirko N. Jan 11 '10 at 20:40
  • As additional source of informaiton netstat -anp --tcp|grep --color can be – zaletniy Sep 12 '11 at 08:31
  • add finally block after try-catch and close all streams. It might be possible that streams are not closing properly and therefor you got this error. Also note that linux treats open connections counts in open files, so check ur code for open connection. – Yogesh Kulkarni May 16 '14 at 13:42
  • See also https://stackoverflow.com/questions/4289447/java-too-many-open-files – Raedwald Mar 14 '19 at 16:15

6 Answers6

26

The problem comes from your Java application (or a library you are using).

First, you should read the entire outputs (Google for StreamGobbler), and pronto!

Javadoc says:

The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.

Secondly, waitFor() your process to terminate. You then should close the input, output and error streams.

Finally destroy() your Process.

My sources:

ofavre
  • 4,488
  • 2
  • 26
  • 23
7

As you are running on Linux I suspect you are running out of file descriptors. Check out ulimit. Here is an article that describes the problem: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

  • 7
    This might get you up to 2 months of run-time! (This is a band-aid at best.) – Greg D Jan 11 '10 at 20:40
  • 1
    You've re-iterated the symptom but not a real cure. – Carl Smotricz Jan 11 '10 at 20:46
  • In fact I did raise the limit in `/etc/security/limits.conf` already for other reasons. – Mirko N. Jan 11 '10 at 20:48
  • This at least can solve our problem. The application (3rd Party) is opening more than 1024. There is NO leak, but just the limit it too low. Setting the limit to 65535 as suggested in this [article](https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/resolve_too_many_open_files_error_and_native_outofmemory_due_to_failed_to_create_thread_issues_in_websphere_application_server_running_on_linux?lang=en). It looks like the problem is gone. – Devs love ZenUML Feb 09 '17 at 06:16
7

Aside from looking into root cause issues like file leaks, etc. in order to do a legitimate increase the "open files" limit and have that persist across reboots, consider editing

/etc/security/limits.conf

by adding something like this

jetty soft nofile 2048
jetty hard nofile 4096

where "jetty" is the username in this case. For more details on limits.conf, see http://linux.die.net/man/5/limits.conf

log off and then log in again and run

ulimit -n

to verify that the change has taken place. New processes by this user should now comply with this change. This link seems to describe how to apply the limit on already running processes but I have not tried it.

The default limit 1024 can be too low for large Java applications.

Community
  • 1
  • 1
Rami Jaamour
  • 171
  • 2
  • 2
5

Don't know the nature of your app, but I have seen this error manifested multiple times because of a connection pool leak, so that would be worth checking out. On Linux, socket connections consume file descriptors as well as file system files. Just a thought.

alasdairg
  • 2,108
  • 12
  • 14
1

You can handle the fds yourself. The exec in java returns a Process object. Intermittently check if the process is still running. Once it has completed close the processes STDERR, STDIN, and STDOUT streams (e.g. proc.getErrorStream.close()). That will mitigate the leaks.

allenjsomb
  • 19
  • 2
-1

This problem comes when you are writing data in many files simultaneously and your Operating System has a fixed limit of Open files. In Linux, you can increase the limit of open files.

https://www.tecmint.com/increase-set-open-file-limits-in-linux/

How do I change the number of open files limit in Linux?

Vpn_talent
  • 1,290
  • 12
  • 21