Running out of memory Java Eclipse

Question

public class Functions {

public Functions (){

}

Map<String, Set<String>> task() throws Exception{

    System.out.println("start reading for strain file");

    String value= "/home/charles/Documents/mmp.txt";
    File file = new File(value);


    FileInputStream fin = new FileInputStream(file);

    FileChannel fc = fin.getChannel();
    MappedByteBuffer mapByteBuff = fc.map(FileChannel.MapMode.READ_ONLY, 0, file.length());

    byte[] buffer= new byte[(int)fc.size()];
    mapByteBuff.get(buffer);
    BufferedReader br=new BufferedReader(new InputStreamReader(new ByteArrayInputStream(buffer)));
    Map<String, Set<String>> strainHashMap = new HashMap<String, Set<String>>(824*12+1,2);


    String line = br.readLine();

    //  String delim="[\t]";
    String[] tokens=new String[3];
    //      String delim="[\t]";

    for (line = br.readLine(); line != null; line = br.readLine()) {

        //          tokens=new String[3];
        tokens=line.split("[\t]",3);
                     //here is where I am stuck.. I get java out of heap memory

        Set<String> strainSet = strainHashMap.get(tokens[1]);
        if(strainSet==null){
            strainSet = new HashSet<String>();
            strainSet.add(tokens[2]);
            strainHashMap.put(tokens[1],strainSet);
        }
        else{
            strainSet.add(tokens[2]);
            strainHashMap.put(tokens[1], strainSet);
        }


        if(strainHashMap.size() % 600000 ==0){
            System.out.println("strain"+strainHashMap.size());
        }


        if(strainHashMap.size() % 610000 ==0){
            System.out.println("strain"+strainHashMap.size());
        }

        if(strainHashMap.size() % 620000 ==0){
            System.out.println("strain"+strainHashMap.size());
        }


        if(strainHashMap.size() % 650000 ==0){
            System.out.println("strain"+strainHashMap.size());
        }

        if(strainHashMap.size() % 700000 ==0){
            System.out.println("strain"+strainHashMap.size());
        }


        if(strainHashMap.size() % 851000 ==0){
            System.out.println("strain"+strainHashMap.size());
        }

    }

    fc.close();
    fin.close();
    System.out.println("The hash stain is " + strainHashMap.size());

    //  System.out.println("The Genotype is " + mapGenotype.size());

    System.out.println("moving out of strain function");
    return strainHashMap;

}

public Map<String, Genotype> genotype() throws Exception {
    // TODO Auto-generated method stub

    System.out.println("entering genotype data function");
    // TODO Auto-generated method stub
    String value= "/home/charles/Documents/mmp.txt";
    File file = new File(value);
    FileInputStream fin = new FileInputStream(file);
    FileChannel fc = fin.getChannel();
    MappedByteBuffer mapByteBuff = fc.map(FileChannel.MapMode.READ_ONLY, 0, file.length()); 

    byte[] buffer= new byte[(int)fc.size()];
    mapByteBuff.get(buffer);

    BufferedReader br=new BufferedReader(new InputStreamReader(new ByteArrayInputStream(buffer)));
    Map<String, Genotype> mapGenotype = new HashMap<String, Genotype>(824*12+1,2);

    String line = br.readLine();
    //  String delim="[\t]";
    String[] tokens=new String[8];
    //  String delim="[\t]";

    for (line = br.readLine(); line != null; line = br.readLine()) {
        tokens=new String[8];
        tokens=line.split("[\t]",8);

        //      tokens=line.split(delim);
        //      Genotype genotypeObject = new Genotype();

        //      genotypeObject.setChromsomeName(tokens[3]);

        //      genotypeObject.setMutation(tokens[6]);

        //      genotypeObject.setPosition(Double.parseDouble(tokens[4]));
        //genotypeObject.setPosition(Double.parseDouble(tokens[4]));

        //      genotypeObject.setRef(tokens[5]);

        if(!mapGenotype.containsKey(tokens[1])){


            Genotype genotypeObject = new Genotype();

            genotypeObject.setChromsomeName(tokens[3]);

            genotypeObject.setMutation(tokens[6]);

            genotypeObject.setPosition(Double.parseDouble(tokens[4]));
            //genotypeObject.setPosition(Double.parseDouble(tokens[4]));

            genotypeObject.setRef(tokens[5]);
            mapGenotype.put(tokens[1], genotypeObject);

        }

        if(mapGenotype.size() % 700000 ==0){
            System.out.println(mapGenotype.size());
        }


        if(mapGenotype.size() % 650000 ==0){
            System.out.println(mapGenotype.size());
        }

        if(mapGenotype.size() % 851000 ==0){
            System.out.println(mapGenotype.size());
        }

    }
    fc.close();
    fin.close();
    System.out.println("leaving Genotype method of function");
    return mapGenotype;

}

}

It is strange that I run out of memory after hash map has added 650000 values in task method. I run a same method with similar file and it works perfectly fine genotype.

I am using jdk 1.6 Ubuntu 12.04, Eclipse kepler. The .ini file has -XX:MaxPermSize=256m -Xms740m -Xmx1412m

The Run as configuration are as below: -Xms912m -Xmx2124m Swap memory is more than 3.5 gb.

I was able to do this task sometime back, but unfortunately, lost all my data and am trying to set all this up again. I have put huge number of hours to fix this, but nothing seems to help.

This is the Error I get: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

score 4 · Answer 1 · edited Oct 15 '13 at 17:03

Your JVM settings appear to be generous to me. I don't see how making them larger will help.

Do I understand your code correctly - do you read the entire file into memory and then loop over all the lines to parse out the entries in the Map?

If yes, I'd rewrite it to only keep only one line in memory at a time: read a line, parse it, add to the Map, repeat.

Run VisualVM to see where memory is being consumed as you run. It might be eye-opening.

score 1 · Accepted Answer · answered Oct 15 '13 at 16:32

1

The most likely explanation is that JVM runs out of memory. For this you need to know that the Xmx setting is the upper limit of what Java is allowed to use no matter what your host might have available.

So you either need to increase the Xmx setting like -Xmx4g and try again, or rewrite your code to use less memory.

answered Oct 15 '13 at 16:32

TwoThe

13,879
6
30
54

He says his runtime config is 2g. – TwoThe Oct 15 '13 at 16:49
I finally had to change my system to fix this out. Probably it were space issues in my Wubi-Ubuntu environment. Not sure though. – Death Metal Oct 25 '13 at 19:41

score 1 · Answer 3 · answered Oct 15 '13 at 16:43

Make sure that the memory parameters you're providing are set in the VM arguments box and not the program arguments box.

I'd suggest printing the value of Runtime.getRuntime().maxMemory() to find out (in bytes) how much memory Java believes it is allowed to use to check that the -Xmx parameter was accepted, and periodically printing the value of Runtime.getRuntime().totalMemory() to see how much memory your data takes and how much is in use just before it crashes.

score 1 · Answer 4 · edited May 23 '17 at 12:05

1

I think you are out of "PermGen" size. Set it like "-XX:MaxPermSize=912m".

Perm space is used to keep informations for loaded classes and few other advanced features like String Pool. Refer : What is perm space?

edited May 23 '17 at 12:05

Community

1
1

answered Oct 15 '13 at 16:49

Kanagavelu Sugumar

18,766
20
94
101

Running out of memory Java Eclipse

4 Answers4