1

I have a web application that hangs under high loads. I'm not going to go into the specifics of the code because I really just want some troubleshooting advice and tooling recommendations.

It's a web app, so each request get's a thread. Under a high load test, the app begins to consume all of the cpu, while becoming unresponsive. I suspect that the request threads are hanging in the new code that we are testing. Due to the fact of the cpu consumption, I'm assuming this must be on my app side. My understanding, which could be wrong, is that total cpu consumption indicated my first troubleshooting efforts should be in looking at the code that's consuming those cycles.

What are some tools and/or methods for inspecting which threads are hanging and on what lines of code? Again, I can easily force the app into the problematic behavior.

I've found and been trying out visualvm. Seems like the perfect tool. Still open for suggestions though. I looked at eclipse TPTP and it seems to be end-of-life-ing as well as requiring a more heavy weight deployment.

chad
  • 7,369
  • 6
  • 37
  • 56
  • Just pause it while it's hung. Look at each thread. You'll see why it's hung. Also `jstack` should give you the same information. You don't need any big tool. – Mike Dunlavey Jul 06 '11 at 01:26
  • @Mike Dunlavey Thanks. That's good advice. Initially, I thought that I would like the lightest weight tool available, which `jstack` certainly qualifies as. While trying to get up to speed on `TPTP`, I abandoned that effort in pursuit of a lighter tool. However, `visualvm` is incredibly light weight for a power tool. Simply install and turn on. All java 1.6 ( update 7 ) and higher apps simply show up in the tool. – chad Jul 07 '11 at 15:21
  • It sounds like `visualvm` is a quality app. Unfortunately, from the doc it appears to [suffer from these common problems](http://stackoverflow.com/questions/1777556/alternatives-to-gprof/1779343#1779343). `jstack` is a crude tool, but it will take you straight to the problem. Other wall-time stack samplers that report percent by line are Zoom, possibly Shark, and possibly oprofile. – Mike Dunlavey Jul 07 '11 at 15:49

2 Answers2

0

You can insert logging messages at starting a thread and closing a thread. Then you start the application and inspect the output while penetrating the code.

Another way is to look for memory leaks. If you are sure you haven't one, you can extend the virtual memory of your JVM.

SvenK
  • 2,584
  • 2
  • 21
  • 24
  • Based upon what I see with tools like `top`, there's no memory issues with the app. The issue is that it consumes all of the cpu. – chad Jul 05 '11 at 18:24
  • Difficult to figure out what it is without looking in your code. Here are some general tips: Try to slim the methods the thread starts, Try to instanciate as few as possible objects, Try to share memory between the threads so that they haven't to compute some results again and again – SvenK Jul 05 '11 at 19:44
0

@chad: do you have Database in whole picture...you may want to start by looking what is happening at DB side...you can very well look into DB locks, current sessions etc.

ag112
  • 5,537
  • 2
  • 23
  • 42
  • Correct me if I'm wrong, but cpu consumption on the application side does not indicate a db issue? In fact, I've also monitored the data tier and network traffic, and that's ruled out. I'll reedit my question to suggest that the data tier has been assumed to be not the issue. – chad Jul 07 '11 at 15:16
  • @chad: in this case, try a jprobe tool for each of your use cases..which can provide you layer wise time consumption.. – ag112 Jul 08 '11 at 05:48