Read large file (Java Heap Space)

Question

I want to read CSV file, create objects from every rows and then save these objects to a database. When i read all lines from my file, and store all objects inside ArrayList i get Java Heap Space Error. I tried to save every record immediately after reading, but then saving records by Hibernate method save() take a lot of time.

I also tried to check size of my arrayList and save data when this size equals 100k (commented part of code).

Question: Is any way to read file partly or better way to store data in Java?

String[] colNames;
String[] values;
String line;
Map<Object1, Object1> newObject1Objects = new HashMap<Object1, Object1>();
Map<Object1, Integer> objIdMap = objDao.createObjIdMap();

    StringBuilder raportBuilder = new StringBuilder();
    Long lineCounter = 1L;
        BufferedReader reader = new BufferedReader(new InputStreamReader(
                new FileInputStream(filename), "UTF-8"));
        colNames = reader.readLine().split(";");
        int columnLength = colNames.length;

        while ((line = reader.readLine()) != null) {
            lineCounter++;
            line = line.replace("\"", "").replace("=", "");
            values = line.split(";", columnLength);

            // Object1
            Object1 object1 = createObject1Object(values);
            if (objIdMap.containsKey(object1)) {
                object1.setObjId(objIdMap.get(object1));
            } else if (newObject1Objects.containsKey(object1)) {
                object1 = newObject1Objects.get(object1);
            } else {
                newObject1Objects.put(object1, object1);
            }

            // ==============================================
            // Object2
            Object2 object2 = createObject2Object(values, object1,
                    lineCounter, raportBuilder);
            listOfObject2.add(object2);
            /*
            logger.error("listOfObject2.size():"+listOfObject2.size());
            if(listOfObject2.size() % 100000 == 0){
                object2Dao.performImportOperation(listOfObject2);
                listOfObject2.clear();
            }
            */
        }
        object2Dao.performImportOperation(listOfObject2);

Maybe use a buffer of 0-50 objects and then save it to the database in one query? (Im unfamilair with hibernate, but most likely that would be possible) It is all about finding the right buffer size, maybe write a method which accepts an int buffer and uses it dynamically. Then depending on your use, you can play with different buffer sizes and find the most optimal one. Since RAM is variable for each computer, there is not a fixed ideal buffer size to optimally insert the data into the database. Being fast is never the case with these amounts of data — engineercoding, Oct 03 '16 at 15:07
which server are you using? you need to increase max heap size. — Rohit Gaikwad, Oct 03 '16 at 15:49

score 1 · Accepted Answer · edited Jan 06 '18 at 09:23

1

Increase of max heap size won't help you if you want to process really large files. Your friend is batching.

Hibernate doesn’t implicitly employ JDBC batching and each INSERT and UPDATE statement is executed separately. Read "How do you enable batch inserts in hibernate?" to get information on how to enable it.

Pay attention to IDENTITY generators, as it disables batch fetching.

edited Jan 06 '18 at 09:23

Vlad Mihalcea

142,745
71
566
911

answered Oct 03 '16 at 17:06

Nikita

6,270
2
24
37

Read large file (Java Heap Space)

1 Answers1