9

everyone,

some weeks ago we changed our background application from Oracle Java 8 to the Adopt JDK 11 (Hotspot). We didn't notice anything in the first weeks and months, but now an update from our data center is pending. In the test system we noticed that there is a memory problem.

We have up to 30 instances of our application running on one server. Each one runs with the Tanuiksoft Service Wrapper and has been allocated different amounts of XMX memory depending on the instance.

Every 10 minutes our service prints the following information

    double total = Runtime.getRuntime().totalMemory() / 1024;
    double max = Runtime.getRuntime().maxMemory() / 1024;
    double free = Runtime.getRuntime().freeMemory() / 1024;

    int activeThreads = Thread.activeCount();
    
    logger.info("Memory: [total={0}] [max={1}] [free={2}] [active threads={3}]", //
            total, max, free, activeThreads);

This is the output from Java 8 where total memory reduces.

If I understand this correctly, the totalMemory is the one the VM lets the operating system reserve - this memory is then no longer available to the operating system. Max memory is understood to be the XMX parameter in the service wrapper. freeMemory is the memory which is still free from the totalMemory.

At that time the output looked like this

Memory: [total=278,016] [max=466,432] [free=183,895] [active threads=33]
Memory: [total=278,016] [max=466,432] [free=176,264] [active threads=33]
Memory: [total=259,072] [max=466,432] [free=132,683] [active threads=33]
Memory: [total=259,072] [max=466,432] [free=125,653] [active threads=33]
Memory: [total=259,072] [max=466,432] [free=118,370] [active threads=33]

You can see that the total memory has been released again. Today with Java 11 it looks like this:

Memory: [total=451,584] [max=524,288] [free=124,309] [active threads=33]
Memory: [total=451,584] [max=524,288] [free=338,183] [active threads=33]

Why doesn't the VM release the memory. It has reserved 451 MB, of which 340 MB are still free. So it would make sense to free the memory again, wouldn't it? The service does not do any further actions until the next restart, so that the memory also goes up and down only minimally. Until the restart (several hours later), the free memory remains at about 300 MB. With Java 8, we have never seen this.

Since some services need a lot of memory at peak times, but it is very unlikely that all services need a lot of memory at the same time, we could live very well with the fact that the sum of all XMX values on the server was always larger than the physical memory. Due to the Java 11 conversion we now have a problem here.

We have also entered the following parameters in the wrapper.conf:

wrapper.java.additional.1=-Xmx512m
wrapper.java.additional.2=-Xms16m
wrapper.java.additional.3=-XX:MaxPermSize=256M
wrapper.java.additional.4=-XX:GCTimeRatio=19
wrapper.java.additional.5=-XX:MinHeapFreeRatio=20
wrapper.java.additional.6=-XX:MaxHeapFreeRatio=30
wrapper.java.additional.7=-XX:-ShrinkHeapInSteps
wrapper.java.additional.8=-Xlog:gc+ergo*=info
wrapper.java.additional.9=-XX:+UseG1GC

We have the same problem with non computing center customers, but they don't notice it so much, because there are only a few instances running and the memory is high enough. In the datacenter the memory is unfortunately somewhat limited.

Does anyone have any idea what we can set here to free the memory?

Hauke
  • 1,405
  • 5
  • 23
  • 44
  • 1
    The garbage collector only perform GC when there's little memory left. In your example you have an amount of memory so there's no point for the garbage collector to GC. – Deadbeef Jul 06 '20 at 12:17
  • Here is something to study: [Introduction to Garbage Collection Tuning](https://docs.oracle.com/en/java/javase/11/gctuning/introduction-garbage-collection-tuning.html#GUID-326EB4CF-8C8C-4267-8355-21AB04F0D304) (Oracle documentation). Also note that permgen space does not exist anymore in newer versions of Java so the `-XX:MaxPermSize` option is not useful anymore. – Jesper Jul 06 '20 at 12:19
  • @Jesper Oh we forgot to remove that - but I think it will skip this parameter, so it souldn't be a problem? – Hauke Jul 06 '20 at 12:52
  • 4
    The main difference between Java 8 and Java 11 from the memory point of view is the use of a different default garbage collection algorithm. Java 11 uses G1 by default. it is better for preventing big pauses on garbage collection, but can use more memory. try turning it off if you need to. – MarianP Jul 06 '20 at 12:53
  • @Enterman That sounds like a problem - so we try to reduce the sum of all xmx values to lower then the physical memory is. But some servers outside the datacenter have enough memory left and the memory isn't released as well. – Hauke Jul 06 '20 at 12:54
  • @MarianP The 9th parameter we just added so see if that helps - but it doesn't – Hauke Jul 06 '20 at 12:54
  • if I understand right you weren't using G1 with Java 8. it is default in Java 11, but not in Java 8 – MarianP Jul 06 '20 at 12:56
  • @MarianP Yes. The parameter wasn't used in Java8 but we tried it today with Java11 - because we hadn't any problems up to now with the memory at all – Hauke Jul 06 '20 at 13:47
  • This article https://openjdk.java.net/jeps/346 explains when Java 11 returns memory. This has been improved in Java 12 – tgdavies Jul 06 '20 at 13:54
  • my point was, it might be the use of G1 that makes your application consume more memory. the parameter XX:+UseG1GC you added doesn't have any effect with Java 11 because G1 is enabled by default there. Java 8 doesn't use G1 by default unless you do turn XX:+UseG1GC on. you can switch the GC used by Java 8 in Java 11 using -XX:+UseParallelGC – MarianP Jul 06 '20 at 13:57
  • did u manage to resolve this? We are seeing memory consumption increase after moving from Oracle Java 8 to AWS Corretto 11. – alext Sep 12 '21 at 21:41

0 Answers0