3

I have project with large database. To parse it I use java with G1 garbage collector. When program runs for a long period of time java begins to consume a lot of memory. But when I check java heap the size is much smaller. For example:

  • Java take 20 Gb of RAM
  • "jmap -histo" - shows me that heap is about 5 Gb of RAM

Question: what is taking the rest of my RAM? Is this overhead of G1?

Edit: Here is stats

RAM stats of my java procces: allocated ~50gb, consumed ~20gb

jmap info: heap size ~4gb

java procces: allocated ~50gb, consumed ~20gb
jmap info: heap size ~4gb

Igor Kiulian
  • 164
  • 8
  • that's how JVM heap works, you specify `-Xmx` and `-Xms` to 20GB and *currently* it takes 5GB only... – Eugene Sep 03 '18 at 10:15
  • @Eugene I'm talking about explicit RAM consumption, -Xmx40G, java consume 20G but heap size is only 4G – Igor Kiulian Sep 03 '18 at 10:18
  • oh, so the process itself consumes 20G and the heap is only 4? how did you measure that 20? – Eugene Sep 03 '18 at 10:19
  • @Eugene i use "htop" to monitor processes, i have added some pictures to show – Igor Kiulian Sep 03 '18 at 10:23
  • So perhaps it *did* use 20Gb of RAM at some point of time? Or it is non-heap memory? – Holger Sep 03 '18 at 12:01
  • @Holget 20Gb is current steate. What can be non-heap memory? What java can store x3 heap size? – Igor Kiulian Sep 03 '18 at 13:40
  • 1
    Yes 20Gb is “current state”. And a current state has a history. When it needed 20Gb at one point, it had to allocate that amount of memory and when it doesn’t need it anymore, i.e. the contained objects have been garbage collected, external tools without knowledge about the Java heap will continue to say that this process has allocated that amount of memory. Whereas inside the JVM, most of the memory is considered free, ready to be filled with new objects. • Off-heap memory can include direct byte buffers. You can have as many and as large, as the available RAM allows. – Holger Sep 04 '18 at 10:26

2 Answers2

4

I understood the problem. As @Holger mentioned the ram is allocated to java process but not fully filled with heap. But the reason why G1 allocates so many ram:

G1 suffers if it needs to allocate a lot of humongous regions. They will be created each time an object size > 50% of the region size. They will waste space as nothing else will be created in the region. Thus if its size is 51%, you will waste 49% of the region. Worse, if a region is 2MB and your object is 2.1MB, it will waste 1.9MB in the second region. If you allocate large objects, adjust your XX:G1HeapRegionSize.

Igor Kiulian
  • 164
  • 8
0

The RAM consumption will be because of the huge database size and the size of the result set.

Try the below: Optimizing Garbage Collection:

  • be wary of String Concatenation operator (+) use concat() instead

  • if using spring, try setFetchSize(number of rows to be fetched at a time) using setFetchSize will, however, increase your time for execution, but it is memory efficient

  • remove all unnecessary statement

  • Use Asynchronous Execution

Anish
  • 155
  • 1
  • 13
  • Thank you for your answer. But i'm not reading whole database at the same time. I use pagination: 1) read small amount of data - chunks; 2) process it; 3) save to db. It should not take so many Gb of RAM. And the most mysterious - the heap is smaller than all memory consumption. The question - what consume this memory if not heap? – Igor Kiulian Sep 03 '18 at 10:33
  • Why do you think that using `concat` has any advantage over using the `+` operator? – Holger Sep 03 '18 at 11:59
  • usually when u specify concat() the compiler knows exactly what to do, ( + ) forces the compiler to understand the context where + is used and + can work with a variety of data types, interpret it and not to mention string concatenation has a big memory hit. you can use concat and toString() together. – Anish Sep 03 '18 at 12:07
  • @Holger see this https://stackoverflow.com/a/8755079/6446770 after reading this, I'm confused how `concat` will be better choice for large values – miiiii Sep 04 '18 at 10:01
  • 1
    @MadMan this is only about a single specific scenario, when you have exactly two items to concatenate and both items are already `String` instances. In that case, `String.concat` may perform better than using a `StringBuilder`, as its implementation creates an array of the right size beforehand and does two plain copy of the input strings into it and creates a new `String` instance with that array (no defensive copy needed). However, JVMs have optimizations specifically to typical `StringBuilder` usage and eliminate the overhead. And Java 9+ will compile the `+` operator entirely different… – Holger Sep 04 '18 at 10:15
  • 1
    @Ani you are confusing the compiler’s work and the runtime performance. All that analysis you mention, will be done at compile time and does not affect the runtime performance. As soon as you combine `String.concat` with `toString()` at the arguments, it’s more expensive than the `+` operator, as these intermediate `String` instances need memory and imply additional copy operations for the character content. I don’t know, why you think they come for free when using `concat`, magic, perhaps? – Holger Sep 04 '18 at 10:20
  • @Holger yup. The same code compiled with Java 8 or below might give some performance overhead, while the same code compiled with JDK9+ will do it better in few sense as the compile time optimization is improved in JDK9 to overcome the JIT overhead, right?? – miiiii Sep 04 '18 at 10:47
  • 1
    @MadMan Starting with Java 9, the string concatenation operator gets compiled to a single `invokedynamic` instruction which gets linked at runtime, hence, it is the JRE which decides, how to implement a particular scenario. So, if you use `+` with exactly two string arguments, it may get linked to some code doing exactly the same as `String.concat`. But even better, *every* particular constellation may get linked to specialized code, generated on-the-fly at runtime. Also, it may implement caches behind the scenes, if considered useful. And so on… – Holger Sep 04 '18 at 11:35