ArrayList Out of Memory Error

Question

I am trying to Retrieve 100,000 records from an SQL database but I am having an Out of Memory Error when using ArrayList.

How can I optimize my code to achieve this? How can I apply a flyweight design pattern to my code?

import java.math.BigDecimal;
import java.util.*;

public class Foo {
    public List<Vo> Service(Vo vo) throws Exception {
        HashMap<Integer, Object> param = null;
        ArrayList<HashMap<String, Object>> getRows = null;
        HashMap<String, Object> row = null;
        Vo Vo = null;

        try {
            List<Vo> list = new ArrayList<Vo>();

            if (Vo != null) {
                param = new HashMap();

                if (Vo.getCode() != null) {
                    param.put(1, Vo.getCode());
                    param.put(2, Vo.getCode());
                }
                if (Vo.getCode() != null) {
                    param.put(3, Vo.getCode());
                    param.put(4, Vo.getCode());
                }

                getRows = (ArrayList) ImpDAO.RetrieveQuery(param);

                if ((getRows != null) && (getRows.size() > 0)) {
                    for (Iterator iterator = getRows.iterator(); iterator.hasNext();) {
                        Vo = new Vo();
                        row = (HashMap) iterator.next();

                        if (row.get("CODE") != null)
                            Vo.setDept((BigDecimal) (row.get("CODE")));
                        if (row.get("SER") != null)
                            Vo.setServCode(row.get("SER").toString());
                        if (row.get("NAME") != null)
                            list.add(Vo);
                    }
                }
            }
        } catch (Exception ae) {
            throw ae;
        }

        return list;
    }
}

Are you confident that the memory savings you would achieve by applying the flyweight pattern would be significant enough to eliminate your memory deficit? Is processing in batches an option? — Tim, Feb 06 '17 at 22:01
flyweight design pattern good in a multi-thread environment but in a single thread environment its not working I think — Narasimha Varma, Feb 07 '17 at 17:34
How much memory do you allow at the moment, what are your system limits? — , Feb 08 '17 at 18:22
Related: https://stackoverflow.com/questions/5066155/arraylist-issue-out-of-memory-error?rq=1 — , Feb 08 '17 at 18:22
`catch (Exception ae) { throw ae;` - that is 100% pointless. — user2357112, Feb 08 '17 at 18:56

score 0 · Answer 1 · edited Feb 08 '17 at 18:47

Double data structures

The biggest problem I see is (ArrayList)ImpDAO.RetrieveQuery(param): RetrieveQuery is already returning a list of hash maps. Thus your data structures are in memory twice, though they may be pointing to the same objects.

Consider changing RetrieveQuery to return an iterator (or even better a stream) backed by the underlying result set. This will prevent you from having to keep two large data structures with the same contents in memory.

Object types

What is the type of row.get("SER")? if the answer isn't String you are likely creating a new string for each row. This would dramatically increase your memory footprint.

What about row.get("CODE")? Does this need to be a BigDecimal? would an int work just as well? If so, you could save a few bytes on each row by switching.

100k is a lot of rows

There are very few reasons to grab 100k rows at the same time. Is this going to a ui for display? if so will you really display 100k rows all at once? perhaps you should look at pagination. Is this going to a processing job? perhaps we can set a limit and process it in chunks.

Asking this kind of question leads one to wonder what the underlying problem is.

score -1 · Answer 2 · answered Feb 08 '17 at 18:30

Can you try by modifying the below portion of the code like this:

Iterator iterator = getRows.iterator();

while(iterator.hasNext()) {
                Vo=new Vo(); 
                row = (HashMap)iterator.next();
                if(row.get("CODE") != null )
                    Vo.setDept((BigDecimal) (row.get("CODE")));
                if(row.get("SER") != null )
                    Vo.setServCode(row.get("SER").toString());
                if(row.get("NAME") != null )

                list.add(Vo);
}

ArrayList Out of Memory Error

2 Answers2

Double data structures

Object types

100k is a lot of rows