12

I'm considering whipping up a script to

  1. run once a minute (or every five minutes)
  2. run jstack against a running JVM in production
  3. parse the jstack output and tally up things I'm interested in
  4. export the results for graphing 24/365 by a centralized Cacti installation on another server

But I've no clue as to how expensive or invasive jstack is on a running JVM. How expensive is it to execute jstack on a running JVM? Am I setting myself up for a world of hurt?

Stu Thompson
  • 38,370
  • 19
  • 110
  • 156
  • Have you considered measuring? – Thorbjørn Ravn Andersen Sep 09 '10 at 15:19
  • Instead of running jstack as a separate process, what about running a simple profiler within the application? It would be more efficient, because no network overhead is needed. I wrote such a tool: http://code.google.com/p/h2database/source/browse/trunk/h2/src/main/org/h2/util/Profiler.java - you can also convert it to a .jsp if needed. – Thomas Mueller Sep 09 '10 at 19:27
  • @Thorbjørn Ravn Andersen : Pardon? – Stu Thompson Sep 10 '10 at 05:12
  • @Thomas Mueller: Thanks, I'll have a look at your tool. But "no network overhead"? I don't think that is really a concern--I've got a fat network, and the output of any of these monitoring approaches won't add up to many bytes on the wire. – Stu Thompson Sep 10 '10 at 05:14
  • This version is APL and doesn't have dependencies: http://svn.apache.org/repos/asf/jackrabbit/sandbox/jackrabbit-j3/src/test/java/org/apache/jackrabbit/j3/Profiler.java – Thomas Mueller Sep 10 '10 at 05:44
  • The question is about stats collection, but the answers are about thread dumps? I presume the OP is talking about using options like `-gcutil` with jstack, and I am wondering the same as OP. – haridsv Jul 18 '14 at 16:16
  • @haridsv Nope, was not using `-gcutil`...I wanted the stack traces for all my running threads. – Stu Thompson Jul 18 '14 at 18:28
  • @StuThompson Sorry, I somehow read `jstack` as `jstat`, now the answers make more sense. – haridsv Jul 19 '14 at 08:24

3 Answers3

5

I know this answer is late to the party but the expensive part of jstack comes from attaching to the debugger interface not generally generating the stack traces with an important exception (and the heap size does not matter at all):

Arbitrary stack traces can be generated on a safe point only or while the thread is waiting (i.e. outside java scope). If the thread is waiting/outside java scope the stack requesting thread will carry the task by doing the stack walk on its own. However, you might not wish to "interrupt" a thread to walk its own stack, esp while it is holding a lock (or doing some busy wait). Since there is no way to control the safe points - that's a risk need to be considered.

Another option compared to jstack avoiding attaching to the debugging interface: Thread.getAllStackTraces() or using the ThreadMXBean, run it in the process, save to a file and use an external tool to poll that file.

Last note: I love jstack, it's immense on production system.

bestsss
  • 11,796
  • 3
  • 53
  • 63
  • @StuThompson, I am inclined to answer that as well: http://stackoverflow.com/questions/3651737/why-the-odd-performance-curve-differential-between-bytebuffer-allocate-and-byt the short answer is that non-direct buffers suck for real io, if you need more elaborated info, i can try... I dont post much and I tend to check profiles of people whose posts have interested me. This very question had no selected answer so I hoped I could enlighten it. – bestsss Jun 12 '12 at 17:30
  • um, i was looking for something more than *"non-direct bufferes suck"*...that's something I've documented very well myself with benchmarks ;) I'm really curious about the discrepancies in the cliff there. – Stu Thompson Jun 12 '12 at 19:16
3

Measure. One of the time variants (/usr/bin/time I believe) has a -p option which allows you to see the cpu resources used:

ravn:~ ravn$ /usr/bin/time -p echo Hello World
Hello World
real         0.32
user         0.00
sys          0.00
ravn:~ ravn$

This means that it took 0.32 seconds wall time, using 0.00 seconds of cpu time in user space and 0.00 seconds in kernel space.

Create a test scenario where you have a program running but not doing anything, and try comparing with the usage WITH and WITHOUT a jstack attaching e.g. every second. Then you have hard numbers and can experiment to see what would give a reasonable overhead.

My hunch would be that once every five minutes is neglectible.

Thorbjørn Ravn Andersen
  • 73,784
  • 33
  • 194
  • 347
  • OK: real 0.54, user 0.51, sys 0.07 on an mildly loaded system. Will try later today. Besides how long it takes, what impact does jstack have on the JVM? An all stop? – Stu Thompson Sep 10 '10 at 07:42
  • Remember to measure your application, not the jstack invocation. – Thorbjørn Ravn Andersen Sep 10 '10 at 11:08
  • It's a web app running in Tomcat. Hmmm...maybe I should run jstack in a loop, and compare the number of invocations per minute with the CPU usage of the Java Tomcat process. Thanks suggesting this and helping think it through. – Stu Thompson Sep 13 '10 at 05:38
0

Depending on the number of threads and the size of your heap, jstack could possibly be quite expensive. JStack is meant for troubleshooting and wasn't designed for stats gathering. It might be better to use some form on instrumentation or expose a JMX interface to get the information you want directly, rather than have to parse stack traces.

dogbane
  • 266,786
  • 75
  • 396
  • 414
  • 1000 threads, <1GB heap. I'm considered jstack because I know it and can implement quickly. JMX, while a viable option, is going to take some edumacation. – Stu Thompson Sep 10 '10 at 05:16