Reduce memory usage in java

Question

I have a web service Java program which reads 13,000,000 dates like '08-23-2016 12:54:44' as strings from database. My developing environment is Java 8, MySQL 5.7 and tomcat 8. I have declare a string array String[] data to store it. I use Guice to inject the initial values of data array to empty. However, the memory usage is still huge. This is my code:

String[] data;//size is 1,000,000
void generateDataWrapper(String params) {
        //read over 13000000 dates string
        ResultSet rs = mySQLCon.readData(params);  
        clearData(data);//set to empty string 
        int index = 0;
        while(rs.next()) {
             data[index++] = rs.getString("date");
             if (index == (size - 1)) {//calculate every 1,000,000 total 13 times 
                 //calculate statistics
                 ... 
                 //reset all to empty string
                 clearData(data);    
                 index = 0;
             } 
        }
}
//mySQLCon. readData function
ResultSet readData(String params) {
        try {
             String query = generateQuery(params);
             Statement postStmt = connection.createStatement();
             ResultSet rs = postStmt.executeQuery(query);
        return rs;
        } catch (Exception e) {
        }
        return null;
}

If I call this function once, the memory is reached 12G, If I call it again, the memory goes to 20G, on the third time the memory will goes to 25G and throw a 'java.lang.OutOfMemoryError: GC overhead limit exceeded' error in com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2174)

This is part of the error message:

java.lang.OutOfMemoryError: GC overhead limit exceeded
    com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2174)
    com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1964)
    com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3316)
    com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463)
    com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3040)
    com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2288)
    com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2681)
    com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2547)
    com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2505)
    com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1370)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    java.lang.reflect.Method.invoke(Unknown Source)

I have changed the garbage collection algorithms to: -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode But it's not helping. I have tried change the data to static variables, still will have this problem. Currently the JVM heap is 8g, the tomcat memory is 24g, however, I don't think increase the memory will solve the problem.

I don't understand why my memory is still increasing every time I call this function, Could someone give me some suggestion?

have you tried memory profiling? or reading the docs of the APIs you're using? basically anything else than giving it more memory? — the8472, Aug 06 '16 at 08:48
This is not a complete program - while I can guess what clearData() and size are doing, those are just guesses. That said, memory profiling as suggested by the8472 might be your best way forward: capture a heap dump when the program runs out of memory, and use a profiling tool to figure out who is holding on to the memory. — Lars, Aug 06 '16 at 08:52
Do you really need to use String to store date? Use LocalDateTime and you may use less memory per object. — Tejas Unnikrishnan, Aug 06 '16 at 08:54

score 3 · Accepted Answer · edited Oct 12 '18 at 19:10

Used resources like a ResultSet have to be closed to release the underlying system-resources. This can be done automatically declaring the resources in a try-block like try (ResultSet resultSet =...).
You can try to fetch only a limited number of rows from database when they are requested from ResultSet and not all of them immediately.
Objects get eligible for garbage collection when they are not referenced any more. So, your array-object keeps in memory with it's whole size as long as it is referenced. If it's not referenced any more and the VM is running out of memory it will be able to dispose the array-object possibly avoiding an OutOfMemoryError.
Unexpectedly high memory usage can be analyzed by creating a heap dump and exploring it in the tool jvisualvm of the JDK.

Qvery · Answer 2 · 2016-08-06T10:03:23.267

Additionally you can change your string array to an long array since strings consume a huge amount of memory. In your case the size of a date string is 38 bytes ( 19 char * 2 bytes ) whereas a long only takes 8 bytes of memory.

long[] data;//size is 1,000,000
void generateDataWrapper(String params) {
    //read over 13000000 dates string
    ResultSet rs = mySQLCon.readData(params);
    clearData(data);//set to empty string
    int index = 0;
    SimpleDateFormat formater =  new SimpleDateFormat("MM-dd-YYYY HH:mm:ss");
    while(rs.next()) {
        try{
            Date date = formater.parse(rs.getString("date"));
            data[index++] = date.getTime();
        }catch(ParseException pe) {
            pe.printStackTrace();
        }
        if (index == (size - 1)) {//calculate every 1,000,000 total 13 times
            //calculate statistics
            ...
            //reset all to empty string
            clearData(data);
            index = 0;
        }
    }
}

Wherever you need your string you can just parse it back with the following

SimpleDateFormat formater =  new SimpleDateFormat("MM-dd-YYYY HH:mm:ss");
Date date = new Date(data[i]);
String dateString = formater.format(date);

I like your idea, when I have a huge data or when I optimize it, I will do this way — Judah Flynn, Aug 12 '16 at 17:48

score 0 · Answer 3 · answered Aug 07 '16 at 04:05

First, thanks for all your suggestions. I have figured this out by reading from mm759 and realized that I forgot to close the ResultSet after I have done reading. After I add rs.close(), every time it takes the same time to finish, although the memory will reach the maximum memory I set.

Reduce memory usage in java

3 Answers3