Play Framework calling controller causes Too many open files error

Question

We use Play Framework 2.3.4 with Java and Hibernate/Spring. It runs fine in production for a couple of days and then suddenly this happens:

java.net.SocketException: Too many open files
    at java.net.Socket.createImpl(Socket.java:460)
    at java.net.Socket.getImpl(Socket.java:520)
    at java.net.Socket.setTcpNoDelay(Socket.java:980)
    at com.mysql.jdbc.StandardSocketFactory.configureSocket(StandardSocketFactory.java:134)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:205)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:297)
    ... 75 more

Then some of these:

java.io.FileNotFoundException: /opt/psldata-api-1.0.0/app/controllers/StatsController.java (Too many open files)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at play.utils.PlayIO$.readFileAsString(PlayIO.scala:63)
    at play.api.Application$$anon$1$$anonfun$input$2.apply(Application.scala:302)
    at play.api.Application$$anon$1$$anonfun$input$2.apply(Application.scala:302)
    at scala.Option.map(Option.scala:145)
    at play.api.Application$$anon$1.input(Application.scala:302)
    at play.api.PlayException$ExceptionSource.interestingLines(PlayException.java:81)
    at views.html.defaultpages.devError$$anonfun$apply$1.apply(devError.template.scala:157)
    at views.html.defaultpages.devError$$anonfun$apply$1.apply(devError.template.scala:142)
    at scala.Option.map(Option.scala:145)
    at views.html.defaultpages.devError$.apply(devError.template.scala:142)
    at play.api.GlobalSettings$$anonfun$devError$1$1.apply(GlobalSettings.scala:141)
    at play.api.GlobalSettings$$anonfun$devError$1$1.apply(GlobalSettings.scala:141)
    at play.api.GlobalSettings$class.onError(GlobalSettings.scala:147)
    at play.core.j.JavaGlobalSettingsAdapter.play$core$j$JavaGlobalSettingsAdapter$$super$onError(JavaGlobalSettingsAdapter.scala:36)
    at play.core.j.JavaGlobalSettingsAdapter$$anonfun$onError$2.apply(JavaGlobalSettingsAdapter.scala:36)
    at play.core.j.JavaGlobalSettingsAdapter$$anonfun$onError$2.apply(JavaGlobalSettingsAdapter.scala:36)
    at scala.Option.getOrElse(Option.scala:120)
    at play.core.j.JavaGlobalSettingsAdapter.onError(JavaGlobalSettingsAdapter.scala:36)
    at play.api.Application$class.handleError(Application.scala:319)
    at play.api.DefaultApplication.handleError(Application.scala:402)
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320)
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320)
    at scala.Option.map(Option.scala:145)
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:320)
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:316)
    at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:344)
    at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:343)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
    at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:46)
    at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
    at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
    at scala.concurrent.Promise$class.complete(Promise.scala:55)
    at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:23)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)

And then some of these:

java.io.FileNotFoundException: /opt/psldata-api-1.0.0/app/controllers/APIController.java (Too many open files)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at play.utils.PlayIO$.readFileAsString(PlayIO.scala:63)
    at play.api.Application$$anon$1$$anonfun$input$2.apply(Application.scala:302)
    at play.api.Application$$anon$1$$anonfun$input$2.apply(Application.scala:302)
    at scala.Option.map(Option.scala:145)
    at play.api.Application$$anon$1.input(Application.scala:302)
    at play.api.PlayException$ExceptionSource.interestingLines(PlayException.java:81)
    at views.html.defaultpages.devError$$anonfun$apply$1.apply(devError.template.scala:157)
    at views.html.defaultpages.devError$$anonfun$apply$1.apply(devError.template.scala:142)
    at scala.Option.map(Option.scala:145)
    at views.html.defaultpages.devError$.apply(devError.template.scala:142)
    at play.api.GlobalSettings$$anonfun$devError$1$1.apply(GlobalSettings.scala:141)
    at play.api.GlobalSettings$$anonfun$devError$1$1.apply(GlobalSettings.scala:141)
    at play.api.GlobalSettings$class.onError(GlobalSettings.scala:147)
    at play.core.j.JavaGlobalSettingsAdapter.play$core$j$JavaGlobalSettingsAdapter$$super$onError(JavaGlobalSettingsAdapter.scala:36)
    at play.core.j.JavaGlobalSettingsAdapter$$anonfun$onError$2.apply(JavaGlobalSettingsAdapter.scala:36)
    at play.core.j.JavaGlobalSettingsAdapter$$anonfun$onError$2.apply(JavaGlobalSettingsAdapter.scala:36)
    at scala.Option.getOrElse(Option.scala:120)
    at play.core.j.JavaGlobalSettingsAdapter.onError(JavaGlobalSettingsAdapter.scala:36)
    at play.api.Application$class.handleError(Application.scala:319)
    at play.api.DefaultApplication.handleError(Application.scala:402)
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320)
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320)
    at scala.Option.map(Option.scala:145)
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:320)
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:316)
    at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:344)
    at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:343)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
    at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:46)
    at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
    at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
    at scala.concurrent.Promise$class.complete(Promise.scala:55)
    at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:23)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)

So basically everything further gives this error, until I restart the application. It seems like a memory leak or something is not closed properly, but I have no idea what it could be. Has anybody had this error before? Thanks.

`lsof | grep | more` and you will see all opened by play proccess files. — MipH, Apr 08 '16 at 10:44

score 1 · Answer 1 · edited May 23 '17 at 10:28

I can see from your file paths that you're using Linux, which has a limit on the number of files and sockets you can open, which is usually setup to 1024. You can count the number of open file descriptors using the lsof command. The issue you're getting is that you're going over the limit. You can run lsof -u <playuser> | wc -l to check this. You can see the limits you currently have setup per user in /etc/security/limits.conf (for at least Ubuntu and RedHat).

If you've got a legitimate reason to have a lot of file handlers or sockets open, then you can increase the limit using the ulimit command (ulimit -n will set the limit to the maximum number the system will support). But if you don't, and you do have a leak, this will just postpone the inevitable crash.

If there isn't a legitimate reason to have so many file handler and sockets open, I would check to make sure you're closing any sockets you create, and make sure you're closing all of those FileInputStreams. Make sure you're closing them properly in 'finally' blocks. Without seeing your controller code, I can't be sure it's a problem with closing the FileInputStreams, but I recommend checking them.

There's a similar question here with a bit more detail on socket limits.

Play Framework calling controller causes Too many open files error

1 Answers1