jps not listing tomcat process and jmap fails to create heap dump

Question

I have running tomcat (9.0.39.0) server, started by user cpappt and running with pid: 1682650. The machine on which this server runs is not bare metal but a VMware machine and the JVM is from OpenJ9

> ps -ef | grep Bootstrap
cpappt   1682650       1  0 Jan01 ?        01:09:58 /srv/jdk11/bin/java
  -Djava.util.logging.config.file=/srv/test/cpappt/tomcat/conf/logging.properties
  -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server 
  -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError
  -Dcrewportalenvironment=test
  -Denvironment=test
  -Doracle.net.tns_admin=/CONFIG/global/ORA/HA
  -Dspring.profiles.active=test,notification-services-intern
  -Djdk.tls.ephemeralDHKeySize=2048
  -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
  -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
  -Djava.awt.headless=true
  -Dcom.sun.management.jmxremote
  -Dfile.encoding=UTF-8
  -Dsun.jnu.encoding=UTF-8
  -Dcom.sun.management.jmxremote.port=7305
  -Dcom.sun.management.jmxremote.ssl=false
  -Dcom.sun.management.jmxremote.password.file=conf/jmxremote.password
  -Dignore.endorsed.dirs= -classpath /srv/test/cpappt/tomcat/bin/bootstrap.jar:/srv/test/cpappt/tomcat/bin/tomcat-juli.jar
  -Dcatalina.base=/srv/test/cpappt/tomcat
  -Dcatalina.home=/srv/test/cpappt/tomcat
  -Djava.io.tmpdir=/srv/test/cpappt/tomcat/temp org.apache.catalina.startup.Bootstrap start

I want to extract a heap dump of the tomcat to analyze a potential memory leak.

I've tried to use jps to acquire the pid of the tomcat server, unfortunately I only see the pid of jps itself.

> jps -l
952152 jdk.jcmd/openj9.tools.attach.diagnostics.tools.Jps

and I'm also running the jps command as the same user that started the tomcat server

> whoami 
cpappt

I've also tried running the jps command, adding the java.io.tmpdir specified in the startup script of the tomcat, which leads to the same result as before (remark: the tmpdir itself is empty).

jps -l -J-Djava.io.tmpdir=/srv/test/cpappt/tomcat/temp

(Additional remark: If I started a small "Foo program", jps is showing me the pid of this program.)

Then I tried to create a heap dump using the pid from the ps command which failed:

> jmap -dump:live,format=b,file=/tmp/dump.hprof 1682650
unrecognized option -dump:live,format=b,file=/tmp/dump.hprof
jmap: obtain heap information about a Java process
 Usage:
    jmap <option>* <vmid>
        <vmid>: Attach API VM ID as shown in jps or other Attach API-based tools
        <vmid>s are read from stdin if none are supplied as arguments
    -histo: print statistics about classes on the heap, including number of objects and aggregate size
    -histo:live : Print only live objects
    -J: supply arguments to the Java VM running jmap
NOTE: this utility might significantly affect the performance of the target VM.
At least one option must be selected.

If I just try to see the histogram, this also fails:

> jmap -histo:live 1682650
Error getting data from 1682650: Exception connecting to 1682650

Also jcmd is not able to create the dump:

> jcmd 1682650 GC.heap_dump /tmp/heapdump
Error getting data from 1682650: Exception connecting to 1682650
...

I somehow suspect that both problems (jps and jmap) are related ... So, if anyone has an idea what might be the problem or even how I could create the heap dump, that would be great.

Additional information from the tomcat server (and java):

Server version number: 9.0.39.0
OS Name:               Linux
OS Version:            5.7.0-0.bpo.2-amd64
Architektur:           amd64
Java Home:             /srv/jdk-11.0.8+10_openj9
JVM Version:           11.0.8+10
JVM Hersteller:        Eclipse OpenJ9

> java -version
openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.21.0, JRE 11 Linux amd64-64-Bit Compressed References 20200715_697 (JIT enabled, AOT enabled)
OpenJ9   - 34cf4c075
OMR      - 113e54219
JCL      - 95bb504fbb based on jdk-11.0.8+10)

> which java
/srv/jdk11/bin/java

> which jps
/srv/jdk11/bin/java

> which jmap 
/srv/jdk11/bin/jmap

Update: 2021-01-09

I finally could get hold of the colleague that knows the password for the jmx user, so using jconsole I was able to trigger a heap dump, but I could only call triggerClassicHeapDump(). I copied the file to my local machine and wanted to analyze it with Eclipse MemoryAnalyzer (I've also added the Diagnostic Tool Framework for Java (DTFJ)).

But when I try to open the file I now get an error message ...

Error opening heap dump 'heapdump.20210119.100934.2621412.0001.txt'. Check the error log for further details.
Error opening heap dump 'heapdump.20210119.100934.2621412.0001.txt'. Check the error log for further details.
Error opening heap dump 'heapdump.20210119.100934.2621412.0001.txt'
Unable to read dump .../heapdump.20210119.100934.2621412.0001.phd metafile .../heapdump.20210119.100934.2621412.0001.txt in DTFJ format DTFJ-PHD (java.io.IOException)
Unable to read dump .../heapdump.20210119.100934.2621412.0001.phd metafile .../heapdump.20210119.100934.2621412.0001.txt in DTFJ format DTFJ-PHD
Error parsing Javacore (java.io.IOException)
Error parsing Javacore
com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException: Maximum line length (32768) exceeded. Input file corrupt or not a javacore. (com.ibm.dtfj.javacore.parser.framework.parser.ParserException)
com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException: Maximum line length (32768) exceeded. Input file corrupt or not a javacore.
Maximum line length (32768) exceeded. Input file corrupt or not a javacore. (com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException)
Maximum line length (32768) exceeded. Input file corrupt or not a javacore.
Unable to read dump .../heapdump.20210119.100934.2621412.0001.txt metafile null in DTFJ format DTFJ-Javacore (java.io.IOException)
Unable to read dump .../heapdump.20210119.100934.2621412.0001.txt metafile null in DTFJ format DTFJ-Javacore
Error parsing Javacore (java.io.IOException)
Error parsing Javacore
com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException: Maximum line length (32768) exceeded. Input file corrupt or not a javacore. (com.ibm.dtfj.javacore.parser.framework.parser.ParserException)
com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException: Maximum line length (32768) exceeded. Input file corrupt or not a javacore.
Maximum line length (32768) exceeded. Input file corrupt or not a javacore. (com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException)
Maximum line length (32768) exceeded. Input file corrupt or not a javacore.

morecore · Answer 1 · 2021-06-23T09:07:32.103

I've added -Xdump:heap:events=user (see Xdump Option Builder) to the startup script of the tomcat and restarted the server.

> ps -ef | grep Bootstrap
cpappt   1919301       1 99 14:38 pts/0    00:01:00 /srv/jdk11/bin/java 
 -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError
 -Xdump:heap:events=user
 ...

Now I can create a heap dump using kill -3 <pid> (have a look at superuser: what-does-kill-3-mean). Two files were created in the catalina.base directory:

javacore.<XXX>.txt (the original thread dump (-3 := SIGQUIT)) and
heapdump.<YYY>.phd (well, the actual heap dump)

Finally I was able to open the heapdump.<YYY>.phd file with Eclipse MAT

Additional remark: We updated the openj9 version to the current version (23.0).

> java -version 
openjdk version "11.0.9" 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9+11)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.23.0, JRE 11 Linux amd64-64-Bit Compressed References 20201022_810 (JIT enabled, AOT enabled)

Now jps also does find the tomcat server

> jps -l
1919301 org.apache.catalina.startup.Bootstrap
1921897 jdk.jcmd/openj9.tools.attach.diagnostics.tools.Jps

Also jcmd does work:

> jcmd 1919301 Dump.heap dump
Dump written to .../tomcat/dump

score 0 · Answer 2 · answered Jun 23 '21 at 08:51

Also consider using -Xdump:system:events=user as the core file generated has a lot more information - however there are problems with the IBM DTFJ reading core files generated by OpenJ9.

Memory Analyzer can also generate heap dumps - see File > Acquire Heap Dump but the 'Attach API' and 'Attach API using a Helper JVM' may require some configuration. Memory Analyzer 1.12 has some improvements in acquiring heap dumps, so try the latest release.

jps not listing tomcat process and jmap fails to create heap dump

2 Answers2