21

I'm trying to debug some issues with java processes on a Solaris box, but running jps returns no output. And jstack gives the error 'Permission denied'. The box is part of a cluster of 3 identical servers, jps and jstack work fine on the 2 other servers.

I found the following forum post from someone with the same problem but no answers: http://forums.sun.com/thread.jspa?threadID=5422237

To clarify running bps and grep for java gives all java proceses correctly, but jps gives nothing (anonymised with 'program' and 'client' to protect the guilty):

program @ clientdelivery2 : ~/
-> bps auxww|grep java
program     3427  5.5 54.067742726649544 ?        S   Sep 25 1039:47 /usr/jdk/instances/jdk1.6.0_16/bin/amd64/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/app/client/program/tomcat/conf/logging.properties -Xmx6144m -XX:PermSize=128m -XX:MaxPermSize=512m -Djava.endorsed.dirs=/app/client/program/tomcat/endorsed -classpath :/app/client/program/tomcat/bin/bootstrap.jar -Dcatalina.base=/app/client/program/tomcat -Dcatalina.home=/app/client/program/tomcat -Djava.io.tmpdir=/app/client/program/tomcat/temp org.apache.catalina.startup.Bootstrap start
program    29915  0.1 11.915252441467896 ?        S 14:55:28  3:59 /usr/jdk/instances/jdk1.6.0_16/bin/amd64/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/app/clientclone/program/tomcat/conf/logging.properties -Xmx2g -XX:PermSize=128m -XX:MaxPermSize=512m -Dcom.sun.management.jmxremote -Djava.endorsed.dirs=/app/clientclone/program/tomcat/endorsed -classpath :/app/clientclone/program/tomcat/bin/bootstrap.jar -Dcatalina.base=/app/clientclone/program/tomcat -Dcatalina.home=/app/clientclone/program/tomcat -Djava.io.tmpdir=/app/clientclone/program/tomcat/temp org.apache.catalina.startup.Bootstrap start
program     1573  0.0  0.0 4760 1332 pts/5    S 17:05:24  0:00 grep --colour java

program @ clientdelivery2 : ~/
-> jps

program @ clientdelivery2 : ~/
->

I asked around and from here http://forums.oracle.com/forums/message.jspa?messageID=5408592 I've got that the issue is:

12460/2: mkdir("/tmp/hsperfdata_program", 0755) Err#13 EACCES [ALL]

Meaning jps is being denied access to the psperfdata directory.

Has anyone run into this issue and know how to solve it?

Sindri Traustason
  • 5,445
  • 6
  • 48
  • 66
  • The java processes are running as the same user that I'm running jps and jstack as. bps is able to list the processes. – Sindri Traustason Sep 27 '10 at 16:03
  • The asker's issue is different, but for me it was because of cron job removing the files in `/tmp/hsperfdata_*` when they are older than 10 days. This resulted in long-running java processes not appearing in `jps` output. See [reference](http://www.semicomplete.com/blog/geekery/jps-shows-nothing-useful.html). – ADTC Feb 20 '14 at 06:53

6 Answers6

20

Turns out the user didn't have access to /tmp because of some issue with mounting the file system. This leads to the files inside hsperfdata_ to never be written, even though the user had access to the /tmp/hsperfdata_ folder itself.

Sindri Traustason
  • 5,445
  • 6
  • 48
  • 66
  • 1
    +1: I had the same issue whereby the /tmp/hsperfdata_ was owned by someone other than the user; someone had chowned it. I had to chown it back to the right user and restart the processes for jps to work correctly. – mogsie Oct 04 '12 at 08:51
  • The asker's issue is different, but for me it was because of cron job removing the files in `/tmp/hsperfdata_*` when they are older than 10 days. This resulted in long-running java processes not appearing in `jps` output. See [reference](http://www.semicomplete.com/blog/geekery/jps-shows-nothing-useful.html). – ADTC Feb 20 '14 at 06:54
  • Similarly under Windows, adding public permissions to `C:\Users\userid\AppData\Local\Temp\hsperfdata_userid` resolved the problem. – df778899 Sep 21 '19 at 17:19
  • I had a file system full in /tmp, cleaning up and restarting the java processes fixed the problem. Thanks! – Olivier Gérardin May 13 '20 at 10:19
8

tldr: sudo jps worked for me (for reasons invoked in other answers)

Thomas
  • 8,306
  • 8
  • 53
  • 92
6

Make sure the program you are trying to 'detect' with jps (and jstack, incidentally) runs without setting the java.io.tmpdir setting, or setting it to the system default.

There are a number of bugs on the Sun Developer Network temp dir locations should not be hardcoded, Fix for 6938627 breaks visualvm monitoring when -Djava.io.tmpdir and Make temporary directory use property java.io.tmpdir which are relevant here.

The story: java Java 6 Update 22 used to use a hardcoded temporary directory for putting data gathered for use by jps and jstack. The jps and jstack programs knew where to look.

However, because someone raised a 'bug' in Java 6 Update 23 they 'fixed' it to use the java.io.tmpdir java runtime setting instead. Now, this defaults to a system-specific location, which is what the 'hardcoded' one was. But if you set the option when invoking your java program, then it will use that instead. Result: jps and jstack look where they expect it to be and don't find anything.

The solution is therefore to ensure that the java.io.tmpdir option is set to the system default (e.g., on the Mac:

> java -Djava.io.tmpdir=$TMPDIR javamain

)

when invoking your program. Then jps and jstack will find it.

My colleague Glyn Normington describes this on his blog. There is apparently a fix in Java 6 Update 25.

Magnus Reftel
  • 967
  • 6
  • 19
Steve Powell
  • 25,354
  • 8
  • 41
  • 44
  • Thanks, particularly for the link to Glyn's post. I was pretty sure I used to be able to see the Tomcat java process locally without a problem. – jabley Apr 01 '11 at 09:50
4

Also, make sure that your startup script does not include -XX:+PerfDisableSharedMem because with this option the JVM will not write any stats, making the process invisible to jps and jstat.

See https://support.datastax.com/hc/en-us/articles/208269876-Java-utilities-such-as-jps-or-jstat-unable-to-monitor-DSE-processes for details.

Bombe
  • 81,643
  • 20
  • 123
  • 127
3

Try:

jps -J-Djava.io.tmpdir=/app/client/program/tomcat/temp
kschneid
  • 5,626
  • 23
  • 31
  • Changes nothing. Why did you suggest that? – Sindri Traustason Sep 29 '10 at 15:28
  • 3
    Because that's where jps looks for `hsperfdata_*` dirs. Do any exist in `/app/client/program/tomcat/temp`? If `java.io.tmpdir` is not set, it would look in `/tmp`. If jps can't find the hsperfdata dir (or if there are no files in that dir), it won't report anything. – kschneid Sep 29 '10 at 15:58
  • 4
    I should probably have added that the args for the two java processes you show include: `-Djava.io.tmpdir=/app/client/program/tomcat/temp` and `-Djava.io.tmpdir=/app/clientclone/program/tomcat/temp`. – kschneid Sep 30 '10 at 13:31
  • 1
    It looks like hsperfdata_ is always put in /tmp whatever java.io.tmpdir is set to. – Sindri Traustason Sep 30 '10 at 16:49
  • 1
    For the sake of follow-up, it looks like what happens is that hotspot will always use the OS's temp dir to create the hsperfdata dirs, but `jps` will allow you to specify `java.io.tempdir` in case you're using a JVM that places those directories in a different location. Sorry about any confusion. – kschneid Oct 01 '10 at 13:12
0

Are you running jps as the same user that is running the Java processes? Even if you run jps as root, it will only return the processes run by that user (root, in this case).

user697547
  • 81
  • 1
  • 5