Can't debug Java Windows Services with jhat, jps, jstack

Question

I frequently showcase the jhat, jps, and jstack tool set to developers on Linux and Mac. However, a developer recently indicated that these are unusable in Windows if the Java app in question is running as a Windows Service.

A Sun-filed bug says something very similar, but was closed due to inactivity.

I have tested this out for myself, and indeed it appears true, though I can hardly believe it. Here is the setup:

1. Tomcat or similar running as a Windows service with the "Log On As" == "Local System"
2. A user with Admin privileges logged in to the same Windows machine.
3. Admin opens Windows Task Manager, can see java.exe running
4. Admin opens console, types "jps", gets back a list of processes that does not include Tomcat's java service process.
5. As a brute force attempt, get the PID of tomcat as a service from Windows Task Manager. Type jstack < pid >. Get a reply: < pid > no such process

This appears reproducible under Windows XP, Windows 2003 Server, and Windows 7. Java versions 1.5 and 1.6 yield the same outcome.

Is there a way from the terminal, even though logged in as Admin, to "sudo up" to get JPS and the other tools to see the java service?

Answer 1

To get the utilities to run, you can connect to the Console session using "mstsc /admin" using a login account (not sure exact permissions it needs to have, mine was in the Administrators group) and use the Sysinternals psexec tool to run as system. Here is an example of using jstack.exe:

psexec -s "%JAVA_HOME%\bin\jstack.exe" PID >stack.txt

Where PID is the process ID of your process. You may also have to substitute the actual path to your JDK depending upon your specific environment.

Also, the TEMP directory must be set correctly or once again the tools will not work.

Answer 2

I've had luck debugging processes run by the SYSTEM user by running this from a command prompt:

c:> time/t
11:18 AM
c:> at 11:19 /interactive cmd.exe
Added a new job with job ID = 1

Use a time 1 minute in the future. When the "at" job runs a windows command prompt will open running as the user SYSTEM. From there you should be able to see the java processes.

If the service is running as a local user (check the details from "mmc %windir%\system32\SERVICES.MSC" by double clicking on the service and selecting the "Log On" tab) you can do the same thing using "runas":

runas /user:USERNAME cmd.exe

Answer 3

You only get those processes that "belong" to you - same user id.

Can you connect to it with jvisualvm?

Answer 4

I've only found a suggest to run visuamvm (or other tools) as windows service: Monitoring Java Processes Running As a Windows Service

Perhaps someone else knows a better solution.

Answer 5

This is my batch file to record a thread dump:

:: Creates a thread dump for the tomcat6.exe process saved in a timestamped filename and views it!
:: Jim Birch 20111128 rev 2015-10-12

::Note this required the following files to be placed in the confluence jre/bin folder:
:: 
:: attach.dll  - From the Java JDK  (must be the same version)
:: tools.jar   - ditto
:: psexec.exe  - from Windows sysinternals
:: Also, the TEMP directory must be set (stack overflow)
set TEMP=c:\windows\temp

::go to run location
d:
cd \confluence.application\jre\bin

::build datetime filename
rem datetime from wmi.exe
for /f "tokens=2 delims==" %%I in ('wmic os get localdatetime /format:list') do set dt0=%%I
rem  datetime string as YYYY-MM-DD-hhmmss
set dt=%dt0:~0,4%-%dt0:~4,2%-%dt0:~6,2%-%dt0:~8,6%
set ff=td-%dt%.txt
echo filename: %ff%

::tomcat PID   
FOR /F "tokens=2" %%I in ('TASKLIST /NH /FI "IMAGENAME eq tomcat6.exe"' ) DO SET PID=%%I
ECHO pid: %PID%

::combine above with jstack command (won't work without psexec)
psexec -s "D:\confluence.application\jre\bin\jstack.exe" -l %PID%  >>  %ff%

:: view output txt file
start %ff%

::pause to review script operation, or use ping to wait a few secs 
::ping localhost -n 20 >nul
pause

Answer 6

Here is the solution that finally worked for me for obtaining stacktraces from unprepared java process running under different user in terminal session or as windows service.

jstack.cmd:

@for /f "usebackq tokens=2" %%I in (`tasklist /NH /FI "imagename eq tomcat.exe" /FI "username eq SYSTEM"`) DO set PID=%%I
@for /f "tokens=2 delims==." %%G in ('wmic os get localdatetime /value') do @set datetime=%%G
@set date_time=%datetime:~0,4%_%datetime:~4,2%_%datetime:~6,2%__%datetime:~8,2%_%datetime:~10,2%_%datetime:~12,2%
PsExec.exe /accepteula -h -s -d cmd /c "%JAVA_HOME%\jstack.exe" -l %PID% ^>"%~dp0jstack_%date_time%.txt"

First line determines the PID by process name and user.

Then PsExec with accepted EULA helps to call jstack in elevated mode with properly escaped redirection of the output to jstack*.txt file in script's dir with timestamp included in file name.

You may need to add PsExec and JDK dirs to PATH beforehand.

I've tried to force threaddumps by SendSignal before but it never worked reliably with Windows service processes.

Answer 7

For GUI based tools like jconsole, please enable JMX support in tomcat, and connect via JMX.

For the command line tools, such as jps, jstat, please run following in an admin console. Note: better add psexec path and %JAVA_HOME%\bin into PATH env.

psexec -S cmd.exe

After we get a new console, we can run the jps, jstat or other tools to "debug" the tomcat running as windows service using local system account.

e:\sde\PSTools>jps
7408 Jps

e:\sde\PSTools>psexec -s cmd

PsExec v2.2 - Execute processes remotely
Copyright (C) 2001-2016 Mark Russinovich
Sysinternals - www.sysinternals.com


Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.

C:\windows\system32>jps
9792 Jps
8812

C:\windows\system32>jstat -gc 8812
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     G
CT
10752.0 10752.0  0.0   10744.3 65536.0  21751.1   175104.0    3730.9   17792.0 17138.5 2176.0 1974.8      1    0.034   0      0.00
0    0.034