java.net.SocketException: java.net.SocketException: Too many open files

Question

I have a large java application (which is used to generate some type of report) in which below class is used to create datasource.

 import org.apache.log4j.Logger;
 import org.springframework.jdbc.datasource.DriverManagerDataSource;
 import com.mysql.jdbc.Connection;
 public class DatabaseConnection
  {
     private static final Logger LOGGER = Logger.getLogger(DatabaseConnection.class.getName());
    @SuppressWarnings("deprecation")
    public static DriverManagerDataSource jdbcConnection(WebmartConfiguration webmartconnection)
    {
        DriverManagerDataSource dataSource = null;
        try
        {
            dataSource = new DriverManagerDataSource("com.mysql.jdbc.Driver", "jdbc:mysql://" + webmartconnection.getHostname() + ":" + webmartconnection.getPort() + "/" + webmartconnection.getDatabasename() + "", webmartconnection.getUsername(), webmartconnection.getPassword());
    }
    catch (Exception sqle)
    {
        LOGGER.info(sqle);
    }
    return dataSource;
 }
}

and that datasource is passed to many method that are used to execute query using JDBCTEMPLATE's query method. For some time application runs smooth and generate reports but after some time application terminated with following stacktrace.

ERROR [run has started] (DivisionThread.java:217) - Could not get JDBC Connection;      nested exception is com.mysql.jdbc.CommunicationsException:     Communications link failure due to underlying exception:    

** BEGIN NESTED EXCEPTION **    

java.net.SocketException    
MESSAGE: java.net.SocketException: Too many open files  

STACKTRACE: 

java.net.SocketException: java.net.SocketException: Too many open files 
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:156)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:276)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2641)
at com.mysql.jdbc.Connection.<init>(Connection.java:1531)
   at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
   at java.sql.DriverManager.getConnection(DriverManager.java:579)
   at java.sql.DriverManager.getConnection(DriverManager.java:190)
   at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:173)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:164)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:149)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:674)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:729)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:745)
at com.mpsinsight.reporting.dao.common.CommonUtilReport.getOutFileName(CommonUtilReport.java:1509)
at com.mpsinsight.reporting.bean.common.ActionEnum$9.getparameterType(ActionEnum.java:507)
at com.mpsinsight.reporting.main.common.JasperReport.populateParameters(JasperReport.java:142)
at com.mpsinsight.reporting.main.common.JasperReport.generateReport(JasperReport.java:61)
at com.mpsinsight.reporting.main.common.GenerateReport$4.generateReport(GenerateReport.java:123)
at com.mpsinsight.reporting.main.common.DivisionThread.run(DivisionThread.java:179)


** END NESTED EXCEPTION **

Example : Below is the one of the method of application that uses datasource.

public String getOutFileName(DriverManagerDataSource datasource, WebmartConfiguration     webmartconnection, String abbrev)
    {
    DriverManagerDataSource dmDatasource = null;
    dmDatasource = datasource;
    if (dmDatasource == null)
    {
        dmDatasource = DatabaseConnection.jdbcConnection(webmartconnection);
    }
    JdbcTemplate jdbcTemplateOb = new JdbcTemplate(dmDatasource);
    String sql = QueryList.Value("outFileNameQuery");

    LOGGER.info("abbrev :::: " + abbrev);
    LOGGER.info("outFileNameQuery :::: " + sql);

    String pathname = (String) jdbcTemplateOb.queryForObject(sql, new Object[] { abbrev }, String.class);

    LOGGER.info("pathname :::: " + pathname);
    return pathname;
}

Could someone explain what could the reason for too many open files error and how can i resolve it.

EDIT : I am working on linux environment. The above error occur only in linux environment.

You're leaking connections, and you don't need to keep recreating the `DataSource` either. One will do. — user207421, Dec 09 '14 at 07:36
Where is your application deployed ? it is linux enviroment ? — Yagnesh Agola, Dec 09 '14 at 08:27
@EJP i am creating connection again only if `dmDatasource` is null. and can you be more clear what you want to say? — Lovish, Dec 09 '14 at 08:30
We can't see from this code where you are creating or closing `Connections` at all. Only `DataSources.` You seem to have them confused. They aren't the same thing. — user207421, Dec 09 '14 at 08:48
The fact that the error occurs when attempting to open a connection to MySQL does not mean that the problem is in the code that makes the connection, your application is leaking resources somewhere, check your code for that. — vzamanillo, Dec 09 '14 at 09:19
As others already mentioned it: There seems to be a resource leak in your application. That it occurs only on Linux only means that you are reaching a limit on the Linux machine earlier than in other systems. You can query the max number of the entire system using `cat /proc/sys/fs/file-max` or to determine the number of used handles by `cat /proc/sys/fs/file-nr` it displays "the total allocated file handles" and "the number of currently used file handles (with the 2.4 kernel); or the number of currently unused file handles (with the 2.6 kernel)." — andih, Dec 15 '14 at 05:47
Your edit still doesn't show any opening or closing of actual connections and other sockets and files. Nothing much can happen here until you address that. — user207421, Dec 15 '14 at 06:06

score 1 · Answer 1 · answered Dec 15 '14 at 06:04

1

Check open file limit using below linux command.

ulimit -a

and you can provide open file limit in below file through root user.

/etc/security/limits.conf

For example

{username}       soft       nofile   1024
{username}       hard   nofile        65536

answered Dec 15 '14 at 06:04

Snehal Patel

1,282
2
11
25

only this `my_username hard maxlogins 10` entry is in `limits.conf` file for my username. should i add two more you suggested above? – Sumit Kamboj Dec 15 '14 at 06:18

score 1 · Answer 2 · edited May 23 '17 at 12:02

The reason for a "too many open files" is that the application or system is reaching the limit of allowed file handles.

This problem usually occurs after several concurrent users get a connection to the Server. Java opens many files in order to read in the classes required to run your application. High volume applications can use a lot of file descriptors. This could lead to a lack of new file descriptors. Also, each new socket requires a descriptor. Clients and Servers communicate via TCP sockets. Each browser's http request consumes TCP sockets when a connection is established to a Server.

In order to prevent a file descriptor leak you should ensure that all resources like streams, database connections are explicitly closed as soon as they are no longer needed. Even Java manages resources for you: don't rely on the garbage collector to clean up your used resources. i.e. close all streams either within a finally block or use the Java 7 try with resources syntax.

The reason for the limit is that the operating system needs memory to manage manage each open file, and memory is a limited resource. Different platforms have different limits on the number of files that can be open in a single process at one time.

On Linux you can as root user change the maximum of the open files count per process (via ulimit -n) and per system (e.g. echo 800000 > /proc/sys/fs/file-max) or per user using the /etc/security/limits.conf file. For the last option you may need a special user running your application.

Before you change one or more of these values you should check whether your application has a resource leak by monitoring the open files. One possible option on Linux is to use strace.

strace -e trace=open,close,read,write,connect,accept your-command-here

You'll need to use the -o option to put strace's output somewhere other than the console, if the process can print to stderr. If your process forks, you'll also need -f or -ff

Or if you want to connect you to a running process and you know the process id you can use strace -p $MyProcess.

If you search the web you'll find a solution for monitoring the open file descriptors that matches your needs. For example monitor open process files on linux (real-time)

"Java opens many files in order to read in the classes required to run your application" is true but it also closes them immediately: usually they are all in JAR files too, which reduces the load considerably. The basic issue is not the number of connections, it is the obvious resource *leak* that is present. — user207421, Dec 15 '14 at 09:06
@EJP You are right that the class loader closes (should) all resources immediately. In some special circumstances usually when the system is under heavy load it may occur that accessing a class for the first time is the reason that such an Exception occurs which usually does not appear. I also think that you are right with your assessment that there exists a resource leak. But at the moment I don't know enough about the environment, load, ... which causes the Problem — andih, Dec 15 '14 at 09:24

java.net.SocketException: java.net.SocketException: Too many open files

2 Answers2