Jmap can't connect to make a dump

Question

We have an open beta of an app which occasionally causes the heapspace to overflow. The JVM reacts by going on a permanent vacation.

To analyze this I would like to peek into the memory at the point where it failed. Java does not want me to do this. The process is still in memory but it doesn't seem to be recognized as a java process.

The server in question is a debian Lenny server, Java 6u14

/opt/jdk/bin# ./jmap -F -dump:format=b,file=/tmp/apidump.hprof 11175
Attaching to process ID 11175, please wait...
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "gHotSpotVMTypeEntryTypeNameOffset" in any of the known library names (libjvm.so, libjvm_g.so, gamma_g)
at sun.jvm.hotspot.HotSpotTypeDataBase.lookupInProcess(HotSpotTypeDataBase.java:390)
at sun.jvm.hotspot.HotSpotTypeDataBase.getLongValueFromProcess(HotSpotTypeDataBase.java:371)
at sun.jvm.hotspot.HotSpotTypeDataBase.readVMTypes(HotSpotTypeDataBase.java:102)
at sun.jvm.hotspot.HotSpotTypeDataBase.<init>(HotSpotTypeDataBase.java:85)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:568)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:494)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:332)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jmap.JMap.runTool(JMap.java:179)
at sun.tools.jmap.JMap.main(JMap.java:110)
Debugger attached successfully.
sun.jvm.hotspot.tools.HeapDumper requires a java VM process/core!

Answer 1

The solution was very simple. I was running the jmap as root, but I had to run it as the user who started the jvm. I will now go hide my head in shame.

Answer 2

I was running the jmap and the application with the same user and still get the error.

The solution was run that comand before the jmap

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Than is just use jmap and will works fine

jmap -heap 17210

Answer 3

If someone tries to get Heap Dump of Java application in Docker container. This is the only solution that worked for me:

docker exec <container-name> jcmd 1 GC.heap_dump /tmp/docker.hprof

It basically dumps the heap of process with pid=1 using jcmd

See https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html

Answer 4

Future Googlers:

This could also happen if you installed the JDK while the process you're trying to jmap was running.

If that's the case, restart the java process.

Answer 5

Follow the below steps to take the thread and Heap dumps from a docker container

1. Run the below command to bash into the container. Please change the CONTAINER_NAME appropriately

   docker exec -it CONTAINER_NAME bash

2. Then type jps to find the all the Java application details and extract the PID for your application

jps

3. Then run the below command to get the thread dump. Please change the PID appropriately

   jstack PID > threadDump.tdump 
   

4. Then run the below command to get the Heap dump. Please change the PID appropriately

    jmap -dump:live,format=b,file=heapDump.hprof PID 

5. Then exit from the docker container and download the threadDump.tdump and heapDump.hprof from the docker container by running the below command. Please change the CONTAINER_NAME appropriately

 sudo docker cp CONTAINER_NAME:threadDump.tdump .
 sudo docker cp CONTAINER_NAME:heapDump.hprof .

Answer 6

What happens if you just run

./jmap -heap 11175 

And are you sure the application JVM is identical to the JMAP JVM? (same version, etc)

Answer 7

You need to use the jmap that comes with the JVM.

Answer 8

I got the same jmap error on a linux machine that have two different OpenJdks installed. First I installed OpenJDK 1.6 and after that OpenJDK 1.7.

A call of ...

/usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/java -XshowSettings:properties -version

# produce the following output ...
...
java.library.path = /usr/java/packages/lib/amd64
    /usr/lib/x86_64-linux-gnu/jni
    /lib/x86_64-linux-gnu
    /usr/lib/x86_64-linux-gnu
    /usr/lib/jni
    /lib
    /usr/lib
...
java version "1.7.0_65"

With including '/usr/lib' every with OpenJDK 1.7.* started program includes the libraries of the first installed JDK (in my case OpenJDK 1.6.*). So the jmap versions of Java6 and Java7 failed.

After I changed the start for the Java7 programms with included OpenJDK 1.7 libraries ...

/usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/java -Djava.library.path=/usr/lib/jvm/java- \
                  7-openjdk-amd64/jre/lib/amd64/server:/usr/java/packages/lib/amd64: \
                  /usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/ \
                  x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib ...

I was able access proccess with the Java 7 version of the jmap program. But it needs a sudo to run.

Answer 9

I have the same problem, I'm trying to find a memory leak in a process running inside a Docker container. I wasn't able to use jmap, instead I used this:

jcmd <pid> GC.class_histogram 

This gives you a list of the objects in the memory. And from the Oracle documentation:

It is recommended to use the latest utility, jcmd instead of jmap utility for enhanced diagnostics and reduced performance overhead. https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks004.html

Answer 10

1.Execute "Docker ps", will give the container Id of all services and collect the container id foe TSC.
2.Execute "docker exec -it CONTAINER_ID bash" (replace CONTAINER_ID with TSC Container id)
3.Bash will come and then execute the "jps" on bash, that will give you the PID for process(it will be 1 for jar)
4.Execute the "jstack PID > threadDump.tdump"(replace PID with process id received in step 3, it should be 1)
5.Execute the "jmap -dump:format=b,file=heapDump.hprof PID"(replace PID with process id received in step 3, it should be 1)
6.Then we have to exit the bash using "exit" command
7.Execute "sudo docker cp CONTAINER_ID:heapDump.hprof ." from ec2 command line, that will copy the dump file on ec2 machine present working directory.
8.Execute "sudo docker cp CONTAINER_ID:threadDump.tdump ." from ec2 command line, that will copy the dump file on ec2 machine present working directory.

Answer 11

When none of these work or if you don't want to change sensitive OS flags such as ptrace_scope:

Either you can use jconsole/jvisualvm to trigger heap dumps or run any JMX client directly from console as follows as you are doing it locally on the machine that needs the dump and so is faster:

echo 'jmx_invoke -m com.sun.management:type=HotSpotDiagnostic dumpHeap heapdump-20160309.hprof false' | java -jar jmxsh.jar -h $LOCALHOST_OR_IP -p $JMX_PORT

I used the wget https://github.com/davr/jmxsh/raw/master/jmxsh.jar for this example.

Answer 12

What worked for me was to simply issue the command with sudo as in:

sudo jmap -heap 21797

Answer 13

In my case it is not as simple as check the user :(

I have a script called collectd-java which invokes jstat and jmap. I've checked by top that such script is launched, as expected, by the user owning the JVM. However, jstat gives me what I need and jmap can't attach. Here is the script - the echo stuff is just the format I need to present the values:

HOSTNAME="${COLLECTD_HOSTNAME:-localhost}"
INTERVAL="${COLLECTD_INTERVAL:-60}"
MAIN_CLASS="my.fully.qualified.MainClass"
PID=$(pgrep -f ${MAIN_CLASS})

get_jstat_classloaderdata() {
VALUE=`jstat -class $PID 1 1 | awk '{print $1}' | grep -vi loaded`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_loaded\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $2}' | grep -vi bytes`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_bytesload\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $3}' | grep -vi unload`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_unloaded\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $4}' | grep -vi bytes`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_bytesunload\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $5}' | grep -vi time`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_time\" interval=$INTERVAL N:$VALUE"
}

get_jmap_heapdata() {
        VALUE=$(jmap -heap ${PID} | grep MinHeapFreeRatio |awk '{print $3}')
        echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-jmap_minheapfreeratio\" interval=$INTERVAL N:$VALUE"

        VALUE=$(jmap -heap ${PID} | grep   MaxHeapFreeRatio|awk '{print $3}')
        echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-jmap_maxheapfreeratio\" interval=$INTERVAL N:$VALUE"

        VALUE=$(jmap -heap ${PID} | grep   MaxHeapSize|awk '{print $3}')
        echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-jmap_maxheapsize\" interval=$INTERVAL N:$VALUE"
}
##Do it
get_jmap_heapdata
get_jstat_classloaderdata

Jstat succeeds and jmap fails. Does anyone understands it ?

Answer 14

Not sure why a plain "jmap " fails when I docker exec -it into my container running centos7 systemd and a java service, but below jmap options worked for me. Thanks: https://dkbalachandar.wordpress.com/2016/07/05/thread-dump-from-a-docker-container/

[root@b29924306cfe /]# jmap 170 Attaching to process ID 170, please wait... Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 170: Operation not permitted sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 170: Operation not permitted

[root@b29924306cfe /]# jmap -dump:live,format=b,file=heapDump.hprof 170 Dumping heap to /heapDump.hprof ... Heap dump file created