7

The user uploads a huge file consisting of 1 million words. I parse the file and put the each line of the file into a LinkedHashMap<Integer, String>.

I need O(1) access and removal by key. Also, I need to preserve the access order, iterate from any position and sort.

The memory consumption is huge. I enabled Strings deduplication feature which appears in Java 8, but it turns out that the LinkedHashMap consumes most of the memory.

I found that LinkedHashMap.Entry consumes 40 bytes, but there are only 2 pointers - one for the next entry and one for the previous entry. I thought 1 pointer should be 64 bits or 32 bits. Buy if I divide 409,405,320(bytes) by 6,823,422(entries count) I have 60 bytes per entry.

I think I don't need the previous pointer, the next pointer should be enough to keep order. Why does LinkedHashMap consume so much memory? How can I reduce memory consumption?

Instance occurence

Community
  • 1
  • 1
user12384512
  • 3,362
  • 10
  • 61
  • 97
  • is it possible the `Integer` wrapper is using that much extra memory? Maybe an [`int`-based implementation](https://github.com/ggrandes/kvstore/blob/master/src/main/java/org/javastack/kvstore/structures/hash/IntLinkedHashMap.java) could help – Salem Jan 04 '17 at 10:51
  • 1
    @1blustone If you look at the image, you can see that `Integer`s are taking up 16% of the heap. `LinkedHashMap.Entry`s are taking up more than 3x more. I believe the OP wants to know why this should be the case. – Michael Jan 04 '17 at 10:54
  • 1
    You are too quick when browsing sources. that entry inherits from `HashMap.Node` which has 4 more fields, and there are additional [object headers](http://stackoverflow.com/q/26357186) whose size is just implementation detail. – glee8e Jan 04 '17 at 11:46
  • I've made a small test and my entry takes 40 bytes `java.util.LinkedHashMap$Entry 240,034,920 (32.3%) 6,000,873 (24.5%)`. Maybe you have some memory leaks: [(https://hoangx281283.wordpress.com/2012/11/18/wrong-use-of-linkedhashmap-causes-memory-leak/](https://hoangx281283.wordpress.com/2012/11/18/wrong-use-of-linkedhashmap-causes-memory-leak/) – Planck Constant Jan 04 '17 at 12:51

1 Answers1

1

How to reduce memory consumption?

1) Add -XX:+UseCompressedOops flag to your JVM startup.

2) Implement your own version of LinkedHashMap, optimized for your needs. I. e. use primitive int as a key instead of Integer, remove "previous" pointer if you don't need it, etc. Note that copying OpenJDK source might be impossible unless you wish to release your modified hash map implementation under GPLv2 license, because OpenJDK is GPLv2. However you can copy and modify LinkedHashMap implementation from Android Open Source Project, because it is Apache licensed.

leventov
  • 14,760
  • 11
  • 69
  • 98