105

One of my applications hangs under some period of running under load, does anyone know what could cause such output in jstack:

"scheduler-5" prio=10 tid=0x00007f49481d0000 nid=0x2061 waiting on condition [0x00007f494e8d0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006ee117310> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

I am seeing this a lot in jstack output when it hangs.

I heavily using Spring @Async & maps, synchronized maps & ehcache.

What is interesting this only happens on one of app instances. Two others are running perfectly fine. What else I could investigate to get more details in such case?

I found this post https://stackoverflow.com/questions/23992787/parking-to-wait-for-0xd8cf0070-a-java-util-concurrent-locks-abstractqueueds but it is not very useful in my case.

Community
  • 1
  • 1
Konrad Pawlus
  • 1,641
  • 3
  • 14
  • 20
  • 3
    That is a SchedeuledExecutorService which will be `waiting` probably most of the time. You should look else where. The waiting in here does not consume CPU so would not cause further performance problems. – John Vint Jun 16 '14 at 13:31
  • If I were a betting man, I would look at your memory usage and GC occurrence. – John Vint Jun 16 '14 at 13:41
  • Ok, thanks. I enabled GC logging, will see what I can find there. – Konrad Pawlus Jun 17 '14 at 11:02
  • 1
    @JohnVint If you were a betting man, what would you suspect in the memory usage and GC occurance? I have a similar problem – user3607022 Apr 22 '15 at 18:51
  • 4
    @user3607022 `GC Overhead limit reached`. That is a clear indication your app has memory issues. If you see that, that means the old generation is pretty much filled and can't be GC'd any further. – John Vint Apr 22 '15 at 18:54
  • Ok, thanks for the insight. Would that possibly cause the application to hang without throwing anything? – user3607022 Apr 22 '15 at 18:57
  • If it's nearly unresponsive you should see that particular error in the logs. If the thread dump doesn't show anything obvious you can look at the CPU usage. If the memory is reasonable and the CPU is low then it is probably something else. – John Vint Apr 22 '15 at 18:58

3 Answers3

229

unsafe.park is pretty much the same as thread.wait, except that it's using architecture specific code (thus the reason it's 'unsafe'). unsafe is not made available publicly, but is used within java internal libraries where architecture specific code would offer significant optimization benefits. It's used a lot for thread pooling.

So, to answer your question, all the thread is doing is waiting for something, it's not really using any CPU. Considering that your original stack trace shows that you're using a lock I would assume a deadlock is going on in your case.

Yes I know you have almost certainly already solved this issue by now. However, you're one of the top results if someone googles sun.misc.unsafe.park. I figure answering the question may help others trying to understand what this method that seems to be using all their CPU is.

Alex Ciminian
  • 11,398
  • 15
  • 60
  • 94
dsollen
  • 6,046
  • 6
  • 43
  • 84
  • 24
    And it is parked because this thread is part of a thread pool for an `ExecutorService`, and it is waiting for some task to be put on the work queue via methods like `ExecutorService.submit(...)`. – Harold L Mar 16 '16 at 21:51
  • 1
    Could it also indicate it's waiting for a response from a connection? Like a backend call? – Nithin Satheesan Dec 23 '20 at 17:31
  • @NithinSatheesan that seems plausible. best way to know is to look at the stack trace and see who is calling it. – dsollen Dec 23 '20 at 18:14
  • @dsollen The stack trace is same as in the post here. It just starts at Thread.run and ends at Unsafe.park. – Nithin Satheesan Dec 23 '20 at 20:00
8

From the stack trace it's clear that the ThreadPoolExecutor > Worker thread started. It's waiting for the task to be available on the BlockingQueue (DelayedWorkQueue) and to pick the task and execute. So, this thread will be in WAIT status until it gets a SIGNAL from the publisher thread.

Kirby
  • 15,127
  • 10
  • 89
  • 104
Yoga Gowda
  • 357
  • 4
  • 8
4

I had a similar issue, and following previous answers (thanks!), I was able to search and find how to handle correctly the ThreadPoolExecutor terminaison.

In my case, that just fix my progressive increase of similar blocked threads:

  • I've used ExecutorService::awaitTermination(x, TimeUnit) and ExecutorService::shutdownNow() (if necessary) in my finally clause.
  • For information, I've used the following commands to detect thread count & list locked threads:
ps -u javaAppuser -L|wc -l
   
jcmd \`ps -C java -o pid=\` Thread.print >> threadPrintDayA.log
    
jcmd \`ps -C java -o pid=\` Thread.print >> threadPrintDayAPlusOne.log
    
cat threadPrint*.log |grep "pool-"|wc -l
boly38
  • 1,806
  • 24
  • 29