51

I am running Ubuntu server edition and I wanted to take a thread dump of Tomcat.

So, I first tried to find out which PID tomcat uses:

$ jps -l
5809 sun.tools.jps.Jps

But it's not there?

So, I used top instead and found out the PID 5730.

Then I called jstack to get the thread dump:

$ sudo jstack -l 5730
5730: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

What's going on? :-(

I already tried to export CATALINA_TMPDIR as described in Jstack and Jstat stopped working with upgrade to JDK6u23 but that didn't change anything:

$ export CATALINA_TMPDIR=/tmp
$ sudo /etc/init.d/tomcat6 restart
 * Stopping Tomcat servlet engine tomcat6
   ...done.
 * Starting Tomcat servlet engine tomcat6
   ...done.
$ sudo jstack -l 5934 // new PID after restart
5934: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Update:

I also tried sudo -u tomcat6 jstack -l -F 5730 > threaddumpexceptions2.txt but it only gives me tons of exceptions on the console.

Community
  • 1
  • 1
Timo Ernst
  • 15,243
  • 23
  • 104
  • 165
  • 1
    Have you tried the -F option? – Michael Pigg Sep 14 '11 at 17:51
  • @Michael If I use the -F option, then it actually starts the thread dump but the dump itself just contains a lot of error messages: http://dl.dropbox.com/u/17844821/zeug/threaddump.txt and during the dumping process many many exceptions are thrown: http://dl.dropbox.com/u/17844821/zeug/threaddumpexception.txt – Timo Ernst Sep 14 '11 at 20:40
  • @valmar Is that possible that the jdk used by Tomcat and jdk you are using for jps are not the same jdk? Could you double check it? – Clark Bao Sep 18 '11 at 05:26
  • @Clark Mh, how would I do that? – Timo Ernst Sep 19 '11 at 19:38
  • @valmar check this. http://www.howtogeek.com/howto/linux/installing-tomcat-6-on-ubuntu/ Tomcat relys on JAVA_HOME. but your command line, you can type java -version to see if they are the same version and check the java bin path the command are using. – Clark Bao Sep 20 '11 at 02:36
  • @Clark Thanks for your reply! I installed Tomcat simply via `sudo apt-get install tomcat6` (no manual installation via wget). `echo $JAVA_HOME` returns me nothing! (Empty line) `java -version` gives me this: `java version "1.6.0_20" OpenJDK Runtime Environment (IcedTea6 1.9.9) (6b20-1.9.9-0ubuntu1~10.04.2) OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)` Does this make any sense to you? – Timo Ernst Sep 20 '11 at 13:10
  • @valmar Can you try export JAVA_HOME= and then restart the Tomcat and then try the jps again? – Clark Bao Sep 20 '11 at 13:24
  • @ClarkBao Nope, that didn't do the trick. `jps -l` still returns only `2316 sun.tools.jps.Jps`. `echo $JAVA_HOME` gives `/usr/lib/jvm/java-6-openjdk` – Timo Ernst Sep 20 '11 at 21:50
  • @valmar I cannot look into the script of how Tomcat6 start the java process. You may need to debug into it to see which jvm it is really using. – Clark Bao Sep 20 '11 at 23:02
  • @ClarkBao I checked the script and the JDK path seems to be correct or am I missing something? http://dl.dropbox.com/u/17844821/zeug/tomcat_script.txt – Timo Ernst Sep 21 '11 at 15:08
  • @valmar I cannot open the link. But there is another way to get thread dump. Send server a SIGQUIT signal to force a stack dump.Check this. http://stackoverflow.com/questions/7135682/whats-the-proper-action-plans-to-debug-the-dead-lock-issue-if-its-in-the-produc – Clark Bao Sep 22 '11 at 01:04
  • @ClarkBao Is stack dump = thread dump? Because your link only describes how to get a stack dump. Here is a new link to the script btw: http://zeug.grandia2.de/tomcat_script.txt – Timo Ernst Sep 22 '11 at 13:45
  • @ClarkBao I just tried `sudo kill -QUIT 5244` but that returns me nothing (just a blank new command line) and nothing happens (5244 is the PID of tomcat and -QUIT should send a SIGQUIT to that process). – Timo Ernst Sep 22 '11 at 13:55

10 Answers10

77

I got it working by doing two things:

  1. Changed call to: sudo -u tomcat6 jstack -J-d64 -m pid
  2. Replaced OpenJDK with Sun's original sun-6-jdk and sun-6-jre packages

Explanation for part 1: I switched to 64-bit mode, used sudo and run the command as Tomcat user.

Note: Part 2 might not be necessary. For some users it seems like part 1 is enough. In fact, try to add just the sudo command first. It might already do the trick.

Timo Ernst
  • 15,243
  • 23
  • 104
  • 165
  • 33
    Placing `sudo -u ` in front of the command did the trick for me. – Jono Jun 25 '12 at 15:35
  • 4
    By the way - just by running as the same user, I find I eliminate this error - in my case attempting to connect with yourkit, but I believe the same principle applies. Being in Sun JDK vs. OpenJDK, or specifying 64 vs. 32 bit seems unnecessary for me to fix it. – David W Mar 14 '14 at 20:22
  • 1
    Running the jstack command with the same user under which the jvm was running fixed the issue for me. Thanks @DavidW ! – Andy Dufresne Jul 30 '14 at 09:49
  • The -J-d64 from root did the trick for me, I tried every combination, correct user, sudo, root. With and without native frames (-m) but it started working when I gave the -J-d64 flag, I also needed -F to get around the process being unresponsive. – feldoh Oct 06 '15 at 10:15
  • After switching to tomcat's user, the -F flag was actually making it fail for me. – Alkanshel Mar 22 '17 at 21:58
34

I think you need to run jstack as the same user that runs the Tomcat process. Note also that jps only returns processes for the current user. You would get the pid for the Tomcat process by running jps with sudo or as the Tomcat process user.

This bug report may also be useful: https://bugs.launchpad.net/ubuntu/+source/sun-java6/+bug/597098

Michael Pigg
  • 596
  • 3
  • 5
  • I can't switch to tomcat6 user because it's not actually a "real" unix account on the system. So `su tomcat6` doesn't work. I just tried `sudo jps -l` but it still only gives me `6493 sun.tools.jps.Jps` – Timo Ernst Sep 14 '11 at 20:42
  • I also tried `sudo -u tomcat6 jstack -l -F 5730 > threaddumpexceptions2.txt` but it only gives me tons of exceptions on the console: http://dl.dropbox.com/u/17844821/zeug/threaddump2.txt - The threaddump itself looks like this: http://dl.dropbox.com/u/17844821/zeug/threaddumpexceptions2.txt - Doesn't look right to me. – Timo Ernst Sep 14 '11 at 20:57
  • @valmar: try this to become tomcat6: switches to root: `sudo su -` from root you you can switch to any user `su tomcat6` – Grey Panther Sep 21 '11 at 21:47
  • @Cd-MaN No effect. If I do that, it gives me this (the slashes represents newlines since I cannot post newline characters here in comments): `$ whoami // valmar // $ sudo su - // # whoami // root // # su tomcat6 // # whoami // root ` – Timo Ernst Sep 22 '11 at 13:39
  • 2
    @valmar: this probably means that you do't have a shell set for the given user (tomcat6) - or most precisely you have the shell set to something like /bin/false. To solve this, use `su -p tomcat6` from under the root account (this preserves the current shell). – Grey Panther Oct 20 '11 at 07:53
4

@Valmar, I find the same topic post here. Unable to get thread dump? Any ideas why my app blocks?

It seems the workaround is sudo -u tomcat6 kill -3 <pid>.

Community
  • 1
  • 1
Clark Bao
  • 1,743
  • 3
  • 21
  • 39
  • Nop, that didn't do the trick. When I do that, nothing happens. – Timo Ernst Sep 24 '11 at 19:50
  • 1
    if you are using 64bit, you need to use jstack -J-d64 -m pid, also try to add sudu -u tomcat6 . http://download.oracle.com/javase/6/docs/technotes/tools/share/jstack.html – Clark Bao Sep 25 '11 at 11:48
  • Hm, that only gives me a load of exceptions on the console: http://dl.dropbox.com/u/17844821/zeug/console64.txt – Timo Ernst Sep 25 '11 at 12:19
  • 1
    If I were you,I will try to use another jdk not openjdk. Maybe it's due to openjdk. – Clark Bao Sep 26 '11 at 11:23
  • Interesting. I replaced openjdk-6-jre with sun-java6-jre and now jstack seems to work properly. jps still doesn't show tomcat and its PID though. @Clark Thanks for your help so far! If you post your suggestion to replace the JRE as a new, separate answer, I'll mark it as "the right answer". – Timo Ernst Sep 27 '11 at 15:07
3

Try to switch to process user and then use jstack:

sudo -u {process user} jstack > dump

Joshua
  • 689
  • 7
  • 16
1

This also worked for me:

sudo -u tomcat6 kill -3 pid

It looks like nothing happens but when you look in the logs the stacks are there. They look like exceptions if your not expecting them.

Ted Bigham
  • 4,237
  • 1
  • 26
  • 31
1

I find it useful to use something like 'ps -eo pid,user,command | grep java' to find the actual java command being used, then use the directory to find the matching jstack etc.

# ps -eo user,command | grep '[j]ava' | cut -d' ' -f1
someuser /usr/lib/jvm/java/bin/java

# /usr/lib/jvm/java/bin/java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

So its 64-bit, running as 'someuser'. su to that user and run run jstack etc. from that same directory. (ie. /usr/lib/jvm/java/bin/jstack

Useful when you're on a server with various different installations / implementations of Java.

Cameron Kerr
  • 1,725
  • 16
  • 23
1

For Tomcat users having this issue, check your catalina.out log file.

I was having the same problems "22693: Unable to open socket file: target process not responding or HotSpot VM not loaded". I gave up and was trying to find anything about what happened before it locked up, but then there was the jstack output in the log file.

John C
  • 500
  • 3
  • 8
0

I had same problem, but none of below solution worked for me:

jstack <pid>
jstack -J-d64 -m <pid>
sudo -u <user> jstack ...

I finally upgraded JDK from jdk1.6.0_24 to jdk1.7.0_67 and every things worked.

MK Aftab
  • 39
  • 7
0

For those finding this answer six years after it was asked and they are on CentOS 7 note that SELinux will stop jmap from writing the heap dumps as, out of the box, it will deny writing to a socket unless the directory has the type tomcat_tmp_t.

In my case, I write my heap dumps under /usr/share/tomcat/.jmap. This directory is owned by my runtime user.

Therefore to allow jmap to write to this directory:

semanage fcontext -a -t tomcat_tmp_t "/usr/share/tomcat/.jmap(/.*)?"
restorecon /usr/share/tomcat -vR

Which allows us to then run, as our tomcat user:

jmap -dump:format=b,file=/usr/share/tomcat/.jmap/tomcat-`date +%s`.bin <pid>
Ron
  • 254
  • 2
  • 17
0

My problem was, I ran the process by the terminal. Then for testing Its jstack, I paused the process by ctrl+z. The process was not able to respond. To solve this problem, I resumed the process by fg and checked its jstack by another terminal.

Amir Fo
  • 5,163
  • 1
  • 43
  • 51