0

I try to fill an ArrayList from a csv that contain 20000000 line , each line of the csv contain an id ,a string ,a int and a float. This array array list is an array list of Data(the costructor is below)

On the output of netbeans I see this error : Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded. I create this method : 

public void fillArrayFromCsv(String path){
    BufferedReader buffer1;
    String line= "";
    String fieldSplitFrom=",";

    try{
        buffer1= new BufferedReader(new FileReader(path));
        while ((line = buffer1.readLine() ) != null) {

            String[] data= line.split(fieldSplitFrom);

            this.add(new Data(Integer.parseInt(data[0]),data[1],Integer.parseInt(data[2]),Float.parseFloat(data[3])));

        }
    }
    catch(FileNotFoundException e){

    }
    catch (IOException ex){

    }


}

the constructor of data is this :

public Data(int id, String f1, int f2, float f3) {
    this.id = id;
    this.f1 = f1;
    this.f2 = f2;
    this.f3 = f3;
}

I have no idea to fix it , probably I must to enlarge the Memory of the heap ?I run this code on a Macbook 12" with 8gb ram and 1,1 intel M processor, can I have a different result on another pc ?

riccardo_castellano
  • 69
  • 1
  • 7
  • You do need to enlarge to heap size. Look at this answer; https://stackoverflow.com/questions/15313393/how-to-increase-application-heap-size-in-eclipse – Y.Kaan Yılmaz Nov 04 '17 at 18:52

1 Answers1

2

You have 20 million CSV lines with two ints, a float and a String. Each Data object requires 8 bytes for the header, 4 bytes for each int and the float and 4 bytes for the String object reference. That's a total of 24 bytes. Times 20 million is 480Mb of heap space. You need to figure out the average size of your Strings to calculate the space you'll need for those (8 bytes for the header plus the average length times 20 million).

Once you have this number you can set the size of the heap using the -Xmx option on the command line.

When asking a question like this it's also a good idea to include the flags you start your JVM with.

Speakjava
  • 3,177
  • 13
  • 15
  • sorry, I don't know what is "flags you start your JVM with" , anyway after the program run netbeans show me the same error . – riccardo_castellano Nov 05 '17 at 12:23
  • Flags are the options you specify on the command line when you start your application. Basically things that come after you type java and before you type the class that contains main. from NetBeans you can look at the properties for your project and then select run and look at the VM options. – Speakjava Nov 05 '17 at 20:27
  • no, I didn't use any flag. Somebody has some idea to fix this problem ? – riccardo_castellano Nov 06 '17 at 20:32
  • The answer is you need to *add* a flag, i.e. use -Xmx to make your heap bigger. I've explained what you need to do to calculate the size of your dataset. Once you have that, set -Xmx to be that plus some space for other objects. If the rough size of your data is, for example, 3.5Gb use -Xmx4g and see if that works. If not, keep making it bigger until it does. – Speakjava Nov 07 '17 at 06:49