Large Map freezes my program

Question

I have large Map where I store some objects. The Map is large: it has around 200k objects. When I try to run some methods, that require to read map values, the program freezes. When I debug it, it seems that my IDE is 'collecting data' (picture). It has never completed the task.

I have 16GB RAM. What can I do to speed this up?

Hi, do you insert element in the Map during the execution ? What object type are stored in the map ? — Tuco, Oct 03 '17 at 11:53
How much RAM did you allocate to the JVM? Could you monitor RAM usage with the jconsole? (you should consider as well how big the objects in the map are too) — gusto2, Oct 03 '17 at 11:53
Can you try making a test case that demonstrates your problem? — matt, Oct 03 '17 at 12:15
Can you post a snipet of your code? Will help understadning where the problem is. — IllegalArgumentException, Oct 03 '17 at 12:16
Which jvm are you using? Do your object correctly override equals and hashcode? — matt, Oct 03 '17 at 12:25
This could be the result of a large number of collisions. Maybe look at your hashcode and try to improve it so duplicates are not generated. — jwitt98, Oct 03 '17 at 12:49

matt · Answer 1 · 2017-10-03T12:52:33.410

I get performance issues around 61 million elements.

import java.util.*;

public class BreakingMaps{


    public static void main(String[] args){

            int count = Integer.MAX_VALUE>>5;
            System.out.println(count + " objects tested");
            HashMap<Long, String> set = new HashMap<>(count);
            for(long i = 0; i<count; i++){
                Long l = i;
                set.put(l, l.toString());
            }

            Random r = new Random();
            for(int i = 0; i<1000; i++){
                long k = r.nextInt()%count;
                k = k<0?-k:k;
                System.out.println(set.get(k));
            }

        }

}

I run the program with java -Xms12G -Xmx13G BreakingMaps

I suspect your problem is not the map, but circumstances surrounding the map. If I write the same program, but use a class with hashcode colisions then the program cannot handle 200K elements.

static class Key{
    final long l;
    public Key(long l){
        this.l = l;
    }

    @Override
    public int hashCode(){
        return 1;
    }

    @Override
    public boolean equals(Object o){
        if(o!=null && o instanceof Key){
            return ((Key)o).l==l;
        }
        return false;
    }

}

slesh · Answer 2 · 2017-10-03T13:01:14.900

-1

Look at this - as the solution you can increase the heap size for your app:

java -Xmx6g myprogram.

But it's not very good. I'd suggest you to try to rework your data processing approach. Maybe you can apply some filtering before fetching the data to decrease the data size or implement some calculation on database level.

edited Oct 03 '17 at 13:01

answered Oct 03 '17 at 11:53

slesh

1,902
1
18
29

200K elements in a map isn't too much, and probably shouldn't cause such a decrease in performance. – matt Oct 03 '17 at 12:32

Large Map freezes my program

2 Answers2