23

I have a java app which runs just fine (on Ubuntu 10.04) for few hours until it hits "java.net.SocketException: Too many open files". The code for Sender.java can be found here

Is it because I create a new instance of HttpPut and HttpPost for each thread? I'm using apache-commons HTTPClient 4.

Here's the exception log:

java.net.SocketException: Too many open files
    at java.net.Socket.createImpl(Socket.java:414)
    at java.net.Socket.connect(Socket.java:544)
    at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:123)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:133)
    at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
    at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:108)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
    at com.marketplace.io.Sender.doBasicHttpPost(Sender.java:434)
    at com.marketplace.io.Sender.appVisualExists(Sender.java:223)
    at com.marketplace.io.Sender.addVisualToCollection(Sender.java:350)
    at com.marketplace.service.ImageThread.run(ImageThread.java:136)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:636)
Steve Kuo
  • 61,876
  • 75
  • 195
  • 257
Raunak
  • 6,427
  • 9
  • 40
  • 52

5 Answers5

27

"java.net.SocketException: Too many files open"can be seen any Java Server application e.g. Tomcat, Weblogic, WebSphere etc, with client connecting and disconnecting frequently.

Please note that socket connections are treated like files and they use file descriptor, which is a limited resource.

Different operating system has different limits on number of file handles they can manage.

In short, this error is coming because clients are connecting and disconnecting frequently.If you want to handle it on your side, you have two options :

1) Increase number of open file handles or file descriptors per process.

In UNIX based operating system e.g. Ubuntu or Solaris, you can use command ulimit -a to find out how many open file handles per process is allowed.

$ ulimit -a
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 256
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 2048
virtual memory        (kbytes, -v) unlimited

You can see that, open files (-n) 256, which means only 256 open file handles per process is allowed. If your Java program, remember Tomcat, weblogic or any other application server are Java programs and they run on JVM, exceeds this limit, it will throw java.net.SocketException: Too many files open error.

You can change this limit by using ulimit -n to a larger number e.g. 4096, but do it with advise of UNIX system administrator and if you have separate UNIX support team, than better escalate to them.

2) Reduce timeout for TIME_WAIT state in your operating system

In UNIX based systems, you can see current configuration in /proc/sys/net/ipv4/tcp_fin_timeout file.

In Windows based system, you can see this information in windows registry. You can change the TCPTIME_WAIT timeout in Windows by following below steps :

1) Open Windows Registry Editor, by typing regedit in run command window
2) Find the key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\tcpip\Parameters
3) Add a new key value pair TcpTimedWaitDelay asa decimal and set the desired timeout in seconds (60-240)
4) Restart your windows machine.
Tung Nguyen
  • 1,486
  • 2
  • 18
  • 13
  • I used root to login and executed these commands. But when I switched to other ordinary user, I found the open files number hadn't been changed. Could I configure the number of open file handles for all users? – niaomingjian Jan 10 '17 at 05:52
  • Verify your setting by: tail -f /etc/security/limits.conf. Please make sure your changes has been saved so that it can be applied for all users – Tung Nguyen Jan 11 '17 at 04:58
  • With verifying the setting by: ulimit -n, it seems it's been changed from 1024 to 4096(after running ulimit -n 4096). But bying tail -f /etc/security/limits.conf, I found there isn't any changes in the file. – niaomingjian Jan 11 '17 at 05:10
  • Please edit limits.conf file using: vi /etc/security/limits.conf Append the two rows below to the file to set soft and hard limits for all users (* means apply for all users): * soft nofile 4096 * hard nofile 10240 – Tung Nguyen Jan 11 '17 at 06:55
  • If I appended these two lines by editing limits.conf file, the result of running ulimit -n was still the preceding value 1024. Is it correct? Should I restart my service after editing this file? – niaomingjian Jan 11 '17 at 08:18
15

On line 438 you get the response as a stream and convert that to a byte array. The InputStream returned by entity.getContent() does not get closed. This could be contributing to the problem. Also, the HttpEntity.consumeContent() is deprecated for related reasons.

Wesley Hartford
  • 426
  • 4
  • 6
  • Thanks for going into the trouble and looking at the source code. I'm going to leave the code running overnight and see if your suggested changes make any difference. – Raunak Apr 13 '11 at 23:34
  • 2
    I'm not sure what fixed the problem; whether it was httpClient.getConnectionManager().shutdown(); or your suggestion. Thanks for your help. – Raunak Apr 14 '11 at 12:11
8

You might also want to check the linux maximum open file limit. This related link is for a bespoke java-based product but it nicely explains the steps required to fix the issue.

dkb
  • 4,389
  • 4
  • 36
  • 54
DrUseful
  • 305
  • 2
  • 11
1

(RESOLVED)

I recently have the same error because the var/log on DataBase Server is Full.

#df -h
S.ficheros            Size  Used Avail Use% Montado en

/dev/cciss/c0d0p3     126G  126G    0G 100% /var


#echo "">/var/log/postgresql/postgresql.log

Now, the error is gone!!!

important: see what to log in postgresql.log, review your postgresql.conf

bye
@_jpgo

Kevin Panko
  • 8,356
  • 19
  • 50
  • 61
Juan
  • 19
  • 1
0

I solve the problem by closing the connection in the finally block。

public  static String postMethod(String jsonStr,String sendUrl){
    String resultJson = "";
    HttpClient httpClient = new HttpClient();
    PostMethod post = new PostMethod(sendUrl);
    post.setRequestHeader("Content-Type","application/x-www-form-urlencoded;charset=utf-8");  
    NameValuePair[] param = { new NameValuePair("message",jsonStr)} ;
    post.setRequestBody(param);
    try {
        httpClient.executeMethod(post);
        resultJson = post.getResponseBodyAsString();
    } catch (HttpException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }finally{
         post.releaseConnection();
         ((SimpleHttpConnectionManager)httpClient.getHttpConnectionManager()).shutdown();
    }
    return resultJson;
}
Roronoa Zoro
  • 1
  • 3