4

We are finally(!) updating our production servers to run on Java 8 JDK (still using Java 7 compiler).

During a soak test the application runs fine for 2 days, then suddenly we are experiencing a severe increase in CPU usage (from 30% to 80%). I do appreciate that the next step is to engage with a profiler, but I am interested to know if there are any obvious reasons why an application can seem to bed down so well for such a long time and then suddenly "go rogue" especially, when the conditions around the soak remain constant.

Reverting to Java 7 JDK fixes this issue so there are no reasons outside of the JDK change (increasing database table size for example).

Using the default GC keeps the inital CPU usage the same as with Java 7 for the first two days, changing the GC (to G1) results in immediate and unacceptable high CPU usage from the start of the soak test.

David
  • 589
  • 6
  • 21
  • 2
    *"but I am interested to know if there are any obvious reasons why an application can seem to bed down so well for such a long time and then suddenly "go rogue" especially"* - about a thousand reasons. You seem to have proof that the GC has something to do with it, I've had a similar situation where one Microservice running in a docker instance was doing crazy CPU things while basically being idle, while other microservices doing the same things and running in the same docker images were working just fine. Changing the GC settings made the problem go away, still have no idea why. – Gimby Dec 19 '19 at 08:21
  • this is consistently reproducible. We have proved that with JDK 7 the soak test completes in 4 days with no CPU churn with several runs but the JDK 8 always spikes (and bizarrely always after at least 2 days of bedding). Upping the maximum memory available to the application does not affect the outcome of this test. And this is not the GC this is the JDK – David Dec 19 '19 at 08:23
  • 2
    Just a side note: it's a good thing that you're finally upgrading. But Java 8 will be 6 years old in a few months. Just to mention it... – ernest_k Dec 19 '19 at 08:30
  • we're hoping to migrate just in time for it to stop being supported! – David Dec 19 '19 at 08:31
  • 1
    Get a thread dump (kill -3) when CPU use is high. – rustyx Dec 19 '19 at 09:19
  • I don’t get what you mean with “we're hoping to migrate just in time for it to stop being supported!”. The free support for commercial use is already over and the extended support will last until 2025. Neither seem to match your migration schedule. And whatever you hope to gain from running on an unsupported JVM, surely is an illusion. Except for dealing with issues, like garbage collection performance, which might have been fixed years ago already. If that’s what you are heading for, you seem to be on the road to success. – Holger Dec 19 '19 at 10:32
  • this is a good point. we are now moving from oracle JDK to open JDK so that change in itself may be the reason for the issues we are experiencing, I may well try the open JDK 7 as a new baseline test – David Dec 19 '19 at 10:35
  • have you tried enabling G1GC with Java 7? what you're describing could be due to different garbage collection, and not because of anything specific to Java 8. [according to this](https://www.oracle.com/technetwork/java/javase/tech/g1-intro-jsp-135488.html): _"The Garbage-First (G1) garbage collector is fully supported in Oracle JDK 7 update 4 and later releases"_ – Kaan Dec 20 '19 at 23:52
  • I could try this, however the best performance I get from java 8 is using the parallel collector which is the one that is active on the java 7 jdk. The only reason I tried G1 on java 8 was to see if performance would improve due to background reading around it being optimal for open jdk – David Dec 23 '19 at 08:29
  • I had a very similar issue when migrating from Java 8 to Java 9. In my case it was definitely a GC - https://stackoverflow.com/questions/46582523/why-there-is-performance-degradation-after-6-hours-of-java-9-g1-work-without-th. However, in your case Java 7 and Java 8 both have the same default collector - Parallel GC. So it could be something else. – Dmitriy Dumanskiy Dec 28 '19 at 21:58
  • we are now going to try running open jdk 7 in the performance environment to rebaseline open jdk vs the oracle version – David Jan 02 '20 at 10:09
  • I'm still no wiser why simply changing the JDK from 7 to 8 (same binaries -compiled against 7) and using the same GC from java 7 to java 8 should result in any kind of performance change unless somebody knows of an actual article or online reference that indicates that open jdk will perform significantly different to Oracle's version – David Jan 02 '20 at 15:37

0 Answers0