We have a Java 11 app running in Tomcat 9.0.37 on an older Windows 2012 Server based Ec2 instance in AWS. The application is very memory intensive and receives a lot of traffic. The max heap is set to 50GB and we see large reclaims through GC (10-15GB) several times per minute. There is a lot of churn in the heap. The memory committed to the Tomcat process regularly exceeds the max heap setting by a large amount. We have verified via jconsole that the actual heap is not exceeding its max setting, however the committed memory for the tomcat process will slowly increase throughout the day. Sometimes, it has 90GB+ allocated to it by the end of the day.
At times, this will cause the total memory available on the instance to nearly be exhausted. At one point, while the memory was nearly exhausted we took a heap dump using jmap. (i.e. jmap -dump:file=<heap-file-name> <pid>). Prior to the heap dump, the tomcat process had 84GB committed to it. Once the heap dump completed, the tomcat process had 56 GB committed to it. The size of the memory shown to be consumed by the heap in jconsole before and after the heap dump was taken was unchanged.
So far we've been unable to determine both why the tomcat process takes up so much memory in addition to the max heap setting and also why doing a heap dump frees so much memory from the tomcat process. I've read that doing a heap dump can cause a "stop the world" garbage collection event, and so it makes sense that memory would be reclaimed. But I don't understand why the heap size remains the same (i.e. it seems like it is reclaiming memory, but if it's doing so via garbage collection why are the heap metrics unaffected)?
