4

Currently we are analyzing a tomcat thread dump. A single thread dump of all threads running at that same time on a tomcat contains the following lines:

...
"soldOutJmsConsumerContainer-1" prio=10 tid=0x00007f8409c14800 nid=0x231 in Object.wait() [0x00007f8403a9f000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at com.tc.object.RemoteObjectManagerImpl.waitUntilRunning(RemoteObjectManagerImpl.java:150)
    at com.tc.object.RemoteObjectManagerImpl.basicRetrieve(RemoteObjectManagerImpl.java:216)
    - locked <0x00007f847612c820> (a com.tc.object.RemoteObjectManagerImpl)
...
"catalina-exec-33" daemon prio=10 tid=0x0000000041bc4000 nid=0x832 in Object.wait() [0x00007f8400f73000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at com.tc.object.RemoteObjectManagerImpl.waitUntilRunning(RemoteObjectManagerImpl.java:150)
    at com.tc.object.RemoteObjectManagerImpl.basicRetrieve(RemoteObjectManagerImpl.java:216)
    - locked <0x00007f847612c820> (a com.tc.object.RemoteObjectManagerImpl)
...
"catalina-exec-109" daemon prio=10 tid=0x0000000041469800 nid=0x1e87 in Object.wait() [0x00007f83f84c1000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at com.tc.object.RemoteObjectManagerImpl.waitUntilRunning(RemoteObjectManagerImpl.java:150)
    at com.tc.object.RemoteObjectManagerImpl.basicRetrieve(RemoteObjectManagerImpl.java:216)
    - locked <0x00007f847612c820> (a com.tc.object.RemoteObjectManagerImpl)

In particular we do not understand

- locked <0x00007f847612c820> (a com.tc.object.RemoteObjectManagerImpl)

In our understanding it says three threads are holding a lock to the same monitor at that time. In our understanding and according to JLS this is not possible.

Is our interpretation of the thread dump correct?

Lars
  • 489
  • 5
  • 12

4 Answers4

10

It looks like all these threads are waiting for condition associated with the monitor, i.e. they called wait() method of that monitor.

When thread calls wait() on the monitor it owns, it temporary releases the monitor and need to reacquire it when returning from wait(). So, you can have multiple threads that used to own a monitor but now are waiting in wait() method.

axtavt
  • 239,438
  • 41
  • 511
  • 482
  • +1 but... Can you explain why concretely this isn't the same as all the other answers? If they're waiting for a condition, released the monitor and are now trying to reacquire it how is this, in the end, not exactly the same as all the other answers? – NoozNooz42 Jan 19 '12 at 13:02
  • @NoozNooz42: Well, it's possible that these threads have been notified and now are waiting to reacquire a monitor, but it's higly unlikely. I want to emphasize the point that it's much more likely that these threads haven't been notified yet, and are waiting to be notified instead. – axtavt Jan 19 '12 at 13:07
  • 2
    @axtavt It is not possible the threads are waiting to reacquire. If they were waiting to reacquire their state would be `BLOCKED` not `WAITING` – Dev Jan 19 '12 at 13:15
2

What means "- locked <0x00007f847612c820> (a com.tc.object.RemoteObjectManagerImpl)".

It means its inside the synchronized block for that lock. It can be with WAITING (in which case another thread can acquire/hold the lock, or RUNNING in which case it is holding the lock.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
1

Those threads are waiting to get the lock, not holding the lock.

Boris Pavlović
  • 63,078
  • 28
  • 122
  • 148
  • -1. Waiting for condition (in `wait()` method) is not the same thing as waiting to get the lock. – axtavt Jan 19 '12 at 12:54
  • I am not sure, if this interpretation is correct. RemoteObjectManagerImpl#basicRetrieve() is synchronized. The stack trace has to be read bottom up. The stack trace says: 1) Within the synchronized method the monitor with the id 0x00007f847612c820 holds a lock. Then it jumps into waitUntilRunning and then there it waits on Object#wait(). Yes, they are waiting on object but all of them passed through the synchronized method. Since Terracotta is involved, we assume it is a bug in Terracotta. So our question is: What means "- locked <0x00007f847612c820> (a com.tc.object.RemoteObjectManagerImpl)". – Lars Jan 19 '12 at 12:59
  • @axtavt A thread is BLOCKED when it can't get a lock. A thread is WAITING only when in the wait() method. http://docs.oracle.com/javase/7/docs/api/java/lang/Thread.State.html – Peter Lawrey Jan 19 '12 at 13:20
  • There is no bug here. Basically the L1s are waiting for the cluster to be up and running. No request will be served until the L1 has finished successfully to connect to the L2... Hope this helps. If for some reason, this doesn't seem to happen, you might want to raise the issue (with more details on versions and exactly what you are doing) on the Terracotta forums... – Alex Snaps Jan 20 '12 at 00:36
0

No different threads cannot hold a lock on the same objects. Somebody else holds the lock and all the threads in the dump are waiting on that lock.

Tomasz Stanczak
  • 12,796
  • 1
  • 30
  • 32