14

there was a similar question asked java-thread-dump-waiting-on-object-monitor-line-not-followed-by-waiting-on, but there was no concrete answer, so I will ask my question in hopes to get more info...

In the following thread dump I see that the thread is in the "WAITING (on object monitor)" state - but there is no line with "waiting on " that would indicate what it is waiting for. How do I interpret this thread stack and find out why (and what resource) this thread is waiting on?

"eventTaskExecutor-50" prio=10 tid=0x0000000004117000 nid=0xd8dd in Object.wait() [0x00007f8f457ad000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at com.tibco.tibjms.TibjmsxLink.sendRequest(TibjmsxLink.java:359)
- locked <0x00007f98cbebe5d8> (a com.tibco.tibjms.TibjmsxResponse)
at com.tibco.tibjms.TibjmsxSessionImp._confirmTransacted(TibjmsxSessionImp.java:2934)
at com.tibco.tibjms.TibjmsxSessionImp._confirm(TibjmsxSessionImp.java:3333)
- locked <0x00007f90101399b8> (a java.lang.Object)
at com.tibco.tibjms.TibjmsxSessionImp._commit(TibjmsxSessionImp.java:2666)
at com.tibco.tibjms.TibjmsxSessionImp.commit(TibjmsxSessionImp.java:4516)
at org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:217)
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:577)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:482)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1102)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:996)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

Locked ownable synchronizers:
- <0x00007f901011ca88> (a java.util.concurrent.ThreadPoolExecutor$Worker)

This thread is one of the listener threads configured to accept messages from the Tibco bus.

thanks!

Marina

Community
  • 1
  • 1
Marina
  • 3,894
  • 9
  • 34
  • 41

2 Answers2

19

It's a peculiarity of HotSpot JVM. When dumping a stack, JVM recovers the wait object from the method local variables. This info is available for interpreted methods, but not for compiled native wrappers.

When Object.wait is executed frequently enough, it gets JIT-compiled.
After that there will be no "waiting on" line in a thread dump.

  1. Since wait() must be called on a synchronized object, most often the wait object is the last locked object in the stack trace. In your case it is

    - locked <0x00007f98cbebe5d8> (a com.tibco.tibjms.TibjmsxResponse)

  2. To prevent Object.wait from being JIT-compiled (and thus making wait info always available) use the following JVM option

    -XX:CompileCommand="exclude,java/lang/Object.wait"
Opal
  • 81,889
  • 28
  • 189
  • 210
apangin
  • 92,924
  • 10
  • 193
  • 247
  • 1
    @YuFengShen What is printed in brackets has nothing to do with the waiting object. 0x00007f8f457ad000 is the address of the last known Java stack frame of the given thread. – apangin Aug 05 '18 at 01:09
  • apangin , may I know why the waiting object address isn't the in the printed top line "Object.wait() [0x00007f8f457ad000] " , that is to say it is at 0x00007f8f457ad000? – YuFeng Shen Aug 05 '18 at 01:09
  • Very usful opinion. I got one good JVM option - (-XX:CompileCommand="exclude,java/lang/Object.wait" ) – Vincent Jia Mar 21 '19 at 07:06
2

This thread is waiting the notification from another thread (thread name is TCPLinkReader, if you look over the full thread dump you should be able to find it) which is created by the TIBCO EMS client library.

The stacktrace shows that the Spring application is trying to commit a session. To commit the session EMS client needs to send some data to server and waiting for the confirmation from server that the session is committed successfully or not.

TCPLinkReader thread is a dedicated thread that EMS client use to receive downstream (from server to client) TCP packets.

If you are seeing this thread lasts for long, there are 2 scenarios:

  • Something wrong on the EMS server side, possibly hung

  • there are some defects in the client library that caused deadlock, so server did send the response back but the TCPLinkReader thread did not notify the caller thread.

Last, post the full thread dump if problem persists.

L. Yan
  • 88
  • 5
  • Surprisingly, I do not see any TCPLinkReader threads in my thread dumps... However, your assessment of the situation seems to be correct, as I also suspect that the issues we were seeing were related tot he communication with the Tibco bus... Thank you fro the information! – Marina Sep 19 '14 at 19:42