Java: Does clear() the big size list help in quick garbage collection?

Question

Load 1.5 Million Records from Database 1

Load 1.5 Million Records from Database 2

List<DannDB> dDb = fromNamedQuery(); //return em.createNamedQuery("").getResultList();
List<LannDB> lDb = fromNamedQuery();

Compare its data.

Update/persist into Database (Using JPA)

and program ends after two hours.

Same iteration happens every third hour and many a times give Out of Memory.

Does following statement work, does object becomes out of scope with this?

 dDb.clear();

  or 

 dDb = null

or what else I can do?

Answer 1

Assuming that your goal is to reduce the occurrence of OOMEs over all other considerations ...

Assigning null to the List object will make the entire list eligible for garbage collection. You then need to create a new (presumably empty) list to replace it.

Calling clear() will have a similar effect¹ to nulling and recreating, though the details will depend on the List implementation. (For example, calling clear() on an ArrayList doesn't release the backing array. It just nulls the array cells.)

If you can recycle an ArrayList for a list of roughly the same size as the original, you can avoid the garbage while growing the list. (But we don't know this is an ArrayList!)

Another factor in your use-case is that:

List<DannDB> dDb = fromNamedQuery();

is (presumably) going to create a new list anyway. That would render a clear() pointless. (Just assign null to dDb, or let the variable go out of scope or be reassigned the new list.)

A final issue is that it is conceivable that the list is finalizable. That could mean that the list object takes longer to delete.

Overall, I can't say which of assigning null and calling clear() will be better for the memory footprint. Or that either of these will make a significant difference. But there is no reason why you can't try both alternatives, and observe what happens.

The only other things I can suggest are:

• Increase the heap size (and the RAM footprint).
• Change the application so that you don't need to hold entire database snapshots in memory. Depending on the nature of the comparison, you could do it in "chunks" or by streaming the records².

The last one is the only solution that is scalable; i.e. that will work with an ever larger number of records. (Modulo the time taken to deal with more records.)

---

Important Notes:

1. Manually running System.gc() is unlikely to help. At best it will (just) make your application slower.
2. Since the real problem is that you are getting OOMEs, anything that tries to get the JVM to shrink the heap by giving memory back to the OS will be counterproductive.

---

^{1 - Similar from the perspective of storage management. Obviously, there are semantic differences between clearing a list and creating a new one; e.g. if some other part of your application has a reference to the original list.
2 - Those of you are old enough will remember the classic way of implementing a payroll system with magnetic tape storage. If you can select from the two data sources in the same key order, you may be able to use the classic approach to compare them. For example, reading two resultsets in parallel.}

Answer 2

In the case of SQL, you can get your two ResultSets and compare their data iteratively. This way, you don't have ro save all your data in first place.
I assume that your data looks like this for demonstration purposes:

String email1	String email2	int someInt
abc@def.ghi	jkl@mno.pqr	1234567
xyz@gmail.com		8901234

To detect a difference between two ResultSets of this database:

boolean equals(ResultSet a, ResultSet b) {
    while(a.next() && b.next()) {
        String aEmail1 = a.getString(1);
        String bEmail1 = b.getString(1);
        if(!aEmail1.equals(bEmail1)) return false;
        String aEmail2 = a.getString(2);
        String bEmail2 = b.getString(2);
        if(!aEmail2.equals(bEmail2)) return false;
        int aSomeInt = a.getInt(3);
        int bSomeInt = b.getInt(3);
        if(aSomeInt!=bSomeInt) return false;
        if(a.isLast()!=b.isLast())
            throw new IllegalArgumentException(
                "ResultSets have different amounts of rows!"
            );
    }
    return true;
}

To set the contents of ResultSet oldData (also its corresponding database connection) to ResultSet newData:

void updateA(ResultSet oldData, ResultSet newData) {
    while(oldData.next() && newData.next()) {
        String newEmail1 = newData.getString(1);
        oldData.updateString(1,newEmail1);
        String newEmail2 = newData.getString(2);
        oldData.updateString(2,newEmail2);
        int newSomeInt = newData.getInt(3);
        oldData.updateInt(3,newSomeInt);
        if(oldData.isLast()!=newData.isLast())
            throw new IllegalArgumentException(
                "ResultSets have different amounts of rows!"
            );
    }
}

You can ofcourse leave out the if(a.isLast()!=newData.isLast)) ... and if(oldData.isLast()!=newData.isLast()) ... if you don't care that the two sets don't have the same amount of rows.

Answer 3

The thing is that, by default, once allocated heap memory size does not shrink (I mean the memory size allocated from the operating system). If your Java application at one time needed 2 GB of RAM it will keep that reserved from the operating system by default.

If you can, try to change the design of your application to not firstly load all data into memory, but only load what you really need to do your work.

If you really need the two big batches at the same time think about using the following Java command line argument: "-XX:+UseAdaptiveSizePolicy", which would make it possible to shrink the heap space after big memory usages.

You can also call the garbage collector via "System.gc();", but that a) does not shrink the allocated heap memory without the suggested command line argument, and b) really, you should not think about this. Java will run it on it's own by time.

Edit: Improved my first explanation a bit.

Answer 4

Best for memory usage would be for the list to not go out of scope. So it would be better (memory wise) to just modify the content one by one, keeping only one temporary entry object instead of a whole other list.

So you could create a getNextFromNamedQuery() and hasNextInNamedQuery() method and set the data at the current index.

e.g.:

int i=0;
while(hasNextInNamedQuery()) {
    if(dDb.size()<=i) dDb.add(getNextFromQuery());
    else dDb.set(i,getNextFromQuery());
    i++;
}