2

I have a standalone Java console program which is using around 80-170 (160 is a typical number at average load) threads in production. Some code are replaced in the project which is functionally fine but the thread number is growing continually after starting the program. I recognized the problem when it is almost crashed the virtual machine. Then it had 30.000 threads.

I would like to find the reason of this behaviour at the production environment so if there is any useful tool it must be a command line program or should able to run from remote host.

I checked it with 'ps huH p | wc -l' command and VisualVm program.Unfortunately VisualVm remote giving information about the thread numbers only, "Thread" tab is not active

I tried to get useful information from jstack command , but I couldn't find what is the new thread who created etc. from

"Timer-459" #1864 daemon prio=10 os_prio=0 tid=0x0bb6c400 nid=0x3c4f in Object.wait() [0x2a980000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at java.util.TimerThread.mainLoop(Timer.java:526)
    - locked <0x4d18bc80> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:505)

or

"pool-133-thread-2" #1877 prio=5 os_prio=0 tid=0x42a2c400 nid=0x3e5e waiting on condition [0x2b118000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x4d18ae80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I don't get what are they.

Could you help me how could I find out what code creates new threads again and again? Any good tool?

Thank you!

Alexander
  • 23
  • 6
  • 1
    *Useful tool:* Debugger. It will show all your threads, and can suspend them so you can see call stack and step through the code to see what it is doing. If you don't want to connect a debugger, just get a [thread dump](https://stackoverflow.com/q/14118385/5221149). --- If you know what the new threads are doing, you should be able to figure out why they are started in the first place. – Andreas Aug 17 '18 at 09:46
  • It's a big, monolithic, multithreading application, difficult to debug. About the thread dump, I updated my description. – Alexander Aug 17 '18 at 11:34
  • I read the the attached answer, I think it is not right here. The question was not about how could I take a thread dump but how could I find who created the threads like above or what patterns should I recognize or just get which threads created in past x seconds. – Alexander Aug 17 '18 at 12:02

2 Answers2

1

The thread monitor view in JProfiler can show you the stack trace at which any thread was created as long as CPU recording was active.

enter image description here

Disclaimer: My company develops JProfiler.

Ingo Kegel
  • 46,523
  • 10
  • 71
  • 102
  • 1
    Thank you! I solved the problem by myself it was programming mistake. But without it I wouldn't have known JProfiler. I tried on my own computer, It hink it is a great stuff! Good you're working on a product like this one. – Alexander Aug 22 '18 at 11:25
0

I would say that jstack is a correct tool for such task. You can understand threads nature by thier stack traces and source code. E.g. at java.util.TimerThread.mainLoop(Timer.java:526) means that your code uses Timer class and its thread is waiting for the next schedule. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) means you use one of ExecutorService implementations and its worker thread is waiting for the next task. Try to analyze how many thread pool threads are alive and why, probably they are the source of your issue.

Mikita Harbacheuski
  • 2,193
  • 8
  • 16