Java Error: Too many open files

Question

I have written a java code which is a wrapper for combining 3 different softwares. I run the code for 100,000 times normally. At each run, different files should be opened, re-written and closed where all happen in a correct convention of try and catch. If I run the code on a Linux server, there would be no problem. However, when I run it on my mac after 10s of 1000, the Error of Too many open files occur and subsequently error in loading file X and Y and so on and the program terminates.

one more thing I would like to add: the code runs for 1000 times, then the setting gets changed and again for 1000 times and this process repeats. so for the first 4-5 times, there would be no problem, while after 6-7 times (means 6000 runs) this error occurs.

description is vague and not backed by code. Are you sure closing the files works as expected and is executed? — jlordo, Dec 04 '12 at 15:47
Why not only allow 100 files (or so) open at a time? Let threads queue up to write files until some files are closed. — xagyg, Dec 04 '12 at 15:48
Possible duplicate of http://stackoverflow.com/q/4289447/741249 or http://stackoverflow.com/q/2272908/741249 — THelper, Jan 02 '13 at 10:14

Stephen C · Accepted Answer · 2013-11-28T09:14:57.520

The most likely explanation is that you have a bug in your Java application that is causing it to leak file open files. The way to avoid the problem is to write your code so that file handles are always closed; e.g.

    // Using the 'try with resource' construct
    try (Reader r = new FileReader(...)) {
        // use the reader
    } catch (IOException ex) {
        // ...
    }

or

    // Using classic try/catch/finally
    Reader r = null;
    try {
        r = new FileReader(...);
        // use the reader
    } catch (IOException ex) {
        // ...
    } finally {
        if (r != null) {
            try {
                r.close();
            } catch (IOException ex) {
                // ignore
            }
        }
    }

The chances are that the program worked on one operating system and not another because the limit on the number of open files is different on the two systems. If the Java GC runs, it will close any unreachable file handles that it finds via the respective handle's finalizer. However, this may not happen in time to prevent the "too many files open" exception.

If your application actually requires a large number of files to be open simultaneously, you will need to lift the limit. This is best done using the ulimit shell built-in, assuming that you are using an UNIX-based operating system.

Note however that there will still be a hard limit that it is not possible to exceed. And I believe that applies for Windows as well.

score 4 · Answer 2 · answered Dec 04 '12 at 15:53

You either need to increase the limit of the number of files you allow your process with ulimit e.g.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 71679
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 2048
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 71679
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

The default open files limit is 2048 on this machine. On another machine the limit is set to 364076 ;)

Or you have to decrease your usage. This is not something Java controls, this is something you must control either way.

Thanks a lot @PeterLawrey. I realized what the problem is. Just one question, shall I just increase the limit in unix shell? would it stay effective while I run the code? — Ramin, Dec 04 '12 at 16:03
You have to increase it in the shell before running the program. Anything run from that shell and anything Java runs would "inherit" that limit. — Peter Lawrey, Dec 04 '12 at 16:05

Java Error: Too many open files

2 Answers2

Linked