Play Framework Stop responding after 2-3 days

Question

I am using play framework and it stops responding after 2-3 days and when I restart server then everything works fine.

Please let me know what I am doing wrong. Thanks

Stack trace:

Caused by: io.netty.channel.ChannelException: Failed to open a socket.
        at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:62)
        at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:72)
        at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:454)
        ... 64 common frames omitted
Caused by: java.net.SocketException: Too many open files
        at sun.nio.ch.Net.socket0(Native Method)
        at sun.nio.ch.Net.socket(Net.java:411)
        at sun.nio.ch.Net.socket(Net.java:404)
        at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)
        at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
        at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:60)
        ... 70 common frames omitted

Answer 1

Looks like you are hitting the ulimit for your user. This is likely a function of some or all of the following:

• Your user having the default ulimit (probably 256 or 1024 depending on the OS)
• The amount/type of activity on your Play application

You can identify what open file handles your Play application by:

• Running lsof -p PID on a *nix OS
• Running something like Filemon (from sysinternals) on Windows

You'll likely see everything on your Play applications' classpath plus any files which your application opens e.g. log files, configuration files. In addition, if you are running on a *nix OS then open sockets will also consume file handles so you might see open file handles relating to database connection pools or indeed anything which your Play application communicates with via sockets.

Once you understand what your application is doing w.r.t open file handles you can consider what to do next. Likely to be one of:

• Change your application such that it opens fewer file handles (and, if on *nix, uses fewer socket connections)
• Change your application such that it closes open file handles when finished with them
• Increase the number of open files allowed for your user by invoking ulimit -n <some number> to increase the limit for your current shell. The number you choose cannot exceed the hard limit configured on your host. You can also change the ulimit permanently, more details here.