3

we have a java8 web application running on tomcat8.5.47 server.we have only 20-60 users sessions per time but most of time up to 600mb uploading files on server.we also use hibernate and c3p0 for manage database connections. we monitored server several days and saw sometimes java reserved ram increased suddenly and garbage collector did not released it.how can we manage this?and is there any way to release reserved ram and prevent tomcat from increasing ram? and also any way to decrease used ram in task manager?

these are our settings:

-XX:MaxPermSize=1g  -XX:+UseG1GC  -XX:+UseStringDeduplication  -XX:MaxHeapFreeRatio=15  -XX:MinHeapFreeRatio=5  -XX:-UseGCOverheadLimit  -Xmn1g  -XX:+UseCompressedOops  -Xms10g  -Xmx56g

and it is an image of profiler when this happened: enter image description here

and it is an image of profiler and also task manager after 2 hours: enter image description here

P.s. we use jprofiler to profile and the green colour shows reserved ram and the blue colour is for used ram.also in second box you can track gc activity and third is for classes and forth shows threads activities and last is for cpu activities.
Thank you all for your answers.

leila s
  • 49
  • 7
  • are you _sure_ that task manager reports RAM usage? i.e.: resident memory? I highly doubt that – Eugene Sep 13 '20 at 17:45
  • I dont know.but the problem is when task manager shows up to 56g of ram the cpu usage also increase and we detect slowness on server.is it normal that these amount increase every hours? – leila s Sep 13 '20 at 18:01
  • There's obviously something I'm missing here, but if you don't want the JVM to allocate a 56GB heap, why do you tell it that it can? – Neil Coffey Sep 13 '20 at 19:19
  • 1
    And if the answer is "because it needs a 56GB heap with 20-60 users", I'd be tempted to solve that problem first! – Neil Coffey Sep 13 '20 at 19:20
  • As I said in comments, we know 56g is huge and also know that something is wrong but we have to do this because when the amount of ram shows on task manager increases also cpu usage goes up and we detect slowness until restarting tomcat.so we have to increase heap size to postpone these restart and then try to resolve problems. – leila s Sep 14 '20 at 03:32

2 Answers2

2

These types of questions are never easy, mainly because to get it "right", the person asking them needs to have some basic understanding of how an OS treats and deals with memory; and the fact that there are different types of memory (at least resident, committed and reserved). I am by far not versatile enough to get this entirely right too, but I keep learning and getting better at this. They mean very different things and some of them are usually irrelevant (I find reserved to be such). You are using windows, as such this, imho is a must watch to begin with.

After you watch that, you need to move to the JVM world and how a JVM process. The heap is managed by a garbage collector, so to shrink some un-used heap - the GC needs to be able to do that. And while, before jdk-12, G1 could do that - it was never very eager to. Since jdk-12, there is this JEP that will return memory back, i.e.: it will un-commit memory back. Be sure to read when that happens, though. Also notice that other collectors like Shenandoah and/or ZGC do it much more often.

Of course, since you disable -UseGCOverheadLimit, you get a huge spike in CPU (GC threads are running like crazy to free space) and of course everything slows down. If I were you, i would enable that one back, let GC fail and analyze GC logs to understand what is going on. 56GB of Heap is a huge number for 20-60 users (this surely looks like a leak?). Notice, that without GC logs, this might be impossible to give a solution to.

P.S. Look at the first screen you shared and notice how there are two colors there: green and blue. I don't know what tool is that, but it looks like green is for "reserved memory" and blue is "used" (this is what used means). But it would be great if you said exactly what those are.

Eugene
  • 117,005
  • 15
  • 201
  • 306
  • Yes your right.i know 56g is huge but we really had no choice to prevent server from getting slow and needs for restarting tomcat per weeks or 4days sometimes.we didnt choose windows for server we had to. Also we do alot for increase the code performance and choose right settings but we need more to do and we know that.we love to know more if you can help us by introducing useful resources.thank you so much. – leila s Sep 13 '20 at 18:58
  • 1
    @leilas I've already presented my thoughts - let it fail and take a heap dump. analyze where and why it fails, I see not other option to be honest. – Eugene Sep 13 '20 at 19:01
1

Java8 doesn't return allocated RAM back to OS even if JVM doesn't need it. For that feature you need to move to another version of JDK. This is JEP for that https://openjdk.java.net/jeps/346 it says that it was delivered in version 12 so I assume JDKs with version after 12 should have that feature. The only way to prevent increasing of reserved memory is to decrease Xmx value. And since you are setting it to 56g I assume you are OK with Tomcat consuming up to 56g of memory. So if you think that it is too much then just decrease that number.

Ivan
  • 8,508
  • 2
  • 19
  • 30
  • 1
    1) of course [it does release memory back](https://stackoverflow.com/questions/59362760/does-g1gc-release-back-memory-to-the-os-even-if-xms-xmx/59377080#59377080). 2) "Java8 doesn't return allocated RAM back to OS" - so... once a process makes some memory resident it never gives it back? that of course is highly inaccurate (and you confuse terminology too). Since you already linked that JEP, you might read it again: ...collector may not return **committed** Java heap memory .... 3) _The only way to prevent increasing of reserved memory_ - it is absolutely fine to increase "reserved memory". – Eugene Sep 13 '20 at 17:52