Why do I get "Too many open files" errors?

Question

I find myself having to explicitly call System.gc() in my Groovy scripts to prevent errors like the one below. Why doesn't the garbage collector do this for me? Is there something I can do to cause it to garbage collect to prevent these errors (maybe JAVA_OPTS)?

Caught: java.util.concurrent.ExecutionException: org.codehaus.groovy.runtime.InvokerInvocationException: java.io.IOException: Cannot run program "ls": java.io.IOException: error=24, Too many open files
    at groovyx.gpars.GParsPool.runForkJoin(GParsPool.groovy:305)
    at UsageAnalyzer$_run_closure2_closure6.doCall(UsageAnalyzer.groovy:36)
    at groovyx.gpars.GParsPool$_withExistingPool_closure1.doCall(GParsPool.groovy:170)
    at groovyx.gpars.GParsPool$_withExistingPool_closure1.doCall(GParsPool.groovy)
    at groovyx.gpars.GParsPool.withExistingPool(GParsPool.groovy:169)
    at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:141)
    at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:117)
    at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:96)
    at UsageAnalyzer$_run_closure2.doCall(<removed>)
    at UsageAnalyzer.run(<removed>)

This stack trace is from a parallel program but it happens in sequential programs as well.

This sounds more like a resource leakage, e.g. you're forgetting to .close files/sockets or the like (and "rely" on the GC to take care of external resources, which you absolutely should not) — nos, Sep 27 '11 at 13:56
Can you post a code snippet showing what exactly you're trying to do? What I imagine is happening is you're not releasing the file pointers, causing the JVM/system to run out of file pointers to assign. — FloppyDisk, Sep 27 '11 at 13:58
My guess is you're not closing the `InputStream` after you read from `'ls'.execute()` — tim_yates, Sep 27 '11 at 14:02
If that were the case, then why does calling System.gc() solve the problem? — dromodel, Sep 27 '11 at 14:17
Unclosed streams are essentially a memory leak which may be detected and resolved by the GC. If you're quickly processing many files you could easily hit a problem before the GC runs on its own however and you get the problem described above. That's not to say that's the only possibility. Oh and hopefully you do know that calling System.gc() doesn't actually guarantee that it will run or do anything, so it's never ever the right solution. — Vala, Sep 27 '11 at 15:35

score 6 · Accepted Answer · answered Sep 27 '11 at 14:14

6

As you're using groovy, you can use the convenient methods such as File.withReader(), File.withWriter(), File.withInputStream(), InputStream.withStream() to ensure resources get closed cleanly. This is less cumbersome than using Java's try .. finally idiom, as there's not need to explicitly call close(), or declare a variable outside the try block.

e.g. to read from a file.

File f = new File('/mumble/mumble/')
f.withReader{ r -> 
   // do stuff with reader here
}

answered Sep 27 '11 at 14:14

winstaan74

1,131
7
11

This would work but only for file operations. The error above is an ExecutionException and there is no with withReader on Process, just withWriter. – dromodel Sep 27 '11 at 14:56
1

well, kinda- but the `ExecutionException` seems to wrap a whole chain of exceptions, of which the innermost is an `IOException: error=24, Too many open files`. A quick google reveals http://stuffthathappens.com/blog/2007/11/28/crash-boom-too-many-open-files/ and also http://stackoverflow.com/questions/2044672/ioexception-too-many-open-files - maybe one of those will help – winstaan74 Sep 27 '11 at 17:01

score 3 · Answer 2 · answered Sep 27 '11 at 14:04

3

Definitely look for any place you open files or streams and make sure you close them. It's often beneficial to wrap them like this:

final InputStream in = ...;
try
{
    // Do whatever.
}
finally
{
    // Will always close the stream, regardless of exceptions, return statements, etc.
    in.close();
}

answered Sep 27 '11 at 14:04

Vala

5,628
1
29
55

1

Or use `withInputStream` in Groovy and it closes it for you outside of the closure. And in java `InputStream.close` can throw an `IOException` which you have to also catch or throw from your function – tim_yates Sep 27 '11 at 14:09
I honestly wasn't aware of the withX method of doing it, thanks @tim_yates. – Vala Sep 27 '11 at 14:44
Also would only work for File operations and the above is an execution exception. – dromodel Sep 27 '11 at 14:57
@balor123 actually posting some code might have helped people target your problem rather than just leaving them to guess from the exception – tim_yates Sep 27 '11 at 15:10

Why do I get "Too many open files" errors?

2 Answers2

Linked