0

I'm using Tomcat 9 on a Centos 7 VPS. This server has run fine for two years, but recently crashes as if it's out of memory. The crash occurs between 1 hour and 24 hours after fresh Tomcat kill and start. I have also restarted my VPS. I removed all app logs, to be sure some limit wasn't being reached. There are no errors in any logs, and I've read through all similar issues going back to 2010.

The Eclipse MemoryAnalyzer seemed hopeful, so I added the HeapDumpOnOutOfMemoryError flag to my setenv.sh as shown:

export JAVA_OPTS="-Xms1500m -Xmx1500m -Xss512m -Xmn1000m -XX:+HeapDumpOnOutOfMemoryError"; echo "setenv.sh has set JAVA_OPTS: $JAVA_OPTS";

I also increased all mem params by 500m. But, no heap file was found in the JVM dir, or any other dir.

I am the only user of the site. And between crashes, I restore the main page as proof that TC is still alive, until the eventual crash.

TC hosts static content, and two WAR files. The crashes still occur whether I remove one, or both, of the WARs. I did update my static pages by changing the copyright year to 2021 (something I usually do in January, but forgot this year). I have pulled the original static files from backup, and it still crashes.

Can anyone offer some other ideas on how to track this down?

Olaf Kock
  • 46,930
  • 8
  • 59
  • 90
BobC
  • 353
  • 2
  • 5
  • 17
  • you say "crashes _as if it's out of memory_" - which implies that the reason for crashing can be anything. Further, many people use "crash" as a generic word for anything that causes the application to not react as expected: Is the process terminated? (you say: "after tomcat is killed and restarted - which implies that the JVM is still running. Is it?) What log entries do you see? And you might want to configure Tomcat in `CATALINA_OPTS`, not in `JAVA_OPTS`, specifically when you provide a lot of memory to the process. – Olaf Kock Jun 29 '21 at 07:31
  • By crash I mean ps -aux | grep tomcat no longer shows a listing for the PID. Yeah, the pkill -9 -f tomcat isn't needed, just something I do out of habit. – BobC Jun 29 '21 at 20:32
  • The JVM is not crashing as far as I know. The only way I know to confirm is to look for a hs_err_pidXXXX.log, which a full system search doesn't find any. I will try `CATALINA_OPTS` as you suggest. Didn't realize high memory makes a difference on how the JVM is setup. – BobC Jun 29 '21 at 20:47
  • Crossreferencing the [JAVA_OPTS vs CATALINA_OPTS](https://stackoverflow.com/a/24734330/13447) aspect. To figure out _if_ you have a memory issue: Connect `jconsole` to your process and monitor it this way. You'll have a view of memory consumption - if you can rule this out, that's one less worry. But you'll have to come up with a different assumption then. – Olaf Kock Jun 30 '21 at 06:14
  • Olaf, I switched to `CATALINA_OPTS` as you suggested (I did find that SO post 11222365, before the switch and I now fully understand the difference). Still working after 26 hours, so I'm going to call that fixed. Would you mind adding it as a solution? I'll select it as the actual solution so that people can more easily find it - many people, as you know, don't browse the comments. Thanks. – BobC Jun 30 '21 at 22:23

1 Answers1

0

In the comments it looked like there's a solution, but I'm sceptical about it:

I've recommended to switch from JAVA_OPTS to CATALINA_OPTS (here's the reason)- but that was just a side note. As much as it makes sense, I wouldn't expect it to fix anything: Your Tomcat ends up running with exactly the same configuration, just from a different environment variable.

The only way where I could imagine it fixing anything (specifically memory related) is when both JAVA_OPTS and CATALINA_OPTS declare non-matching memory parameters and somehow the lower one from both wins. In the end, Tomcat is started with both parameters combined and the JVM will only interpret one of them (don't ask me if it's the first, last or undefined)

My recommended general approach is still in the comments: Start by ruling out one assumption after the other - monitor the memory consumption and garbage collection if you assume that you have a memory problem. There it'll be obvious if you expect 8GB heapspace, but you're running on 1GB. Or a memory leak will be obvious. Once you can rule out memory (because the crash happens when there's plenty of memory available) go after the next assumption.

Not quite concrete... I'm wondering if this answer actually helps someone in hindsight...

Olaf Kock
  • 46,930
  • 8
  • 59
  • 90
  • I have done memory analysis at the Tomcat level and JVM level. The cPanel I use with the VPS has some good tools, and they seem accurate. I know two things without question: 1) My Tomcat setup on the VPS has not changed in 2 years. It started crashing a couple weeks ago. Having made minor static file changes, and starting TC, all tests passed. About 2 hours later TC crashed. I killed TC, restored from backup, started, crashed within 3 hours. 2) I changed `setenv.sh` to use `CATALINA_OPTS`, killed TC, put all changed files back, started TC, still running 38 hours later. Can't explain it. – BobC Jul 01 '21 at 17:42