16

In line with this question: How to get stack trace of a thread

I am wondering if anyone could point to information about interpreting Java thread stacks extracted with jstack.

My situation is that I've got a Java EE 5 application running on GlassFish v2.1.1 that hangs periodically (at least 2 -3 times a day).To get it running again I have to kill the Glassfish process and start the domain again. The application usually gets slower and slower responding until it finally hangs completely. Once it has hung I can't get a thread stack. I've been able to get a thread stack as it gets slower and slower but I can't see to locate the problem or the threads that have hung.

Here is the latest thread dump: http://issues.i2m.dk/attachments/66/threadstack.201103211046.txt

Snippet of thread dump:

"p: thread-pool-1; w: 20" daemon prio=10 tid=0x00007efd18843800 nid=0x5f89 in Object.wait() [0x00007efd03381000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"TP-Processor83" daemon prio=10 tid=0x0000000043179000 nid=0x5f87 runnable [0x00007efd03583000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efe72696090> (a java.io.BufferedInputStream)
    at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607)
    at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:662)

"p: thread-pool-1; w: 19" daemon prio=10 tid=0x00007efd18710800 nid=0x5f86 in Object.wait() [0x00007efd03684000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 18" daemon prio=10 tid=0x0000000042a5a000 nid=0x5f85 in Object.wait() [0x00007efd03785000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 17" daemon prio=10 tid=0x0000000042a57000 nid=0x5f84 in Object.wait() [0x00007efd03886000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 16" daemon prio=10 tid=0x000000004321b000 nid=0x5f83 in Object.wait() [0x00007efd03987000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 15" daemon prio=10 tid=0x0000000043219800 nid=0x5f82 in Object.wait() [0x00007efd03a88000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 14" daemon prio=10 tid=0x00000000429b6000 nid=0x5f81 in Object.wait() [0x00007efd03b89000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"TP-Processor82" daemon prio=10 tid=0x0000000043896800 nid=0x5f80 runnable [0x00007efd03c8a000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efe72829510> (a java.io.BufferedInputStream)
    at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607)
    at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:662)

"Thread-44980" daemon prio=10 tid=0x00007efd202c3800 nid=0x5d74 runnable [0x00007efd067b5000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efd78fc0058> (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

"Thread-44979" daemon prio=10 tid=0x00007efd20d76000 nid=0x5d73 runnable [0x00007efd062b0000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efd7789a4e0> (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

"TP-Monitor" daemon prio=10 tid=0x00007efd2139f800 nid=0x5b1 in Object.wait() [0x00007efd0d453000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d46d6c0> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:561)
    - locked <0x00007efd4d46d6c0> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:662)

"TP-Monitor" daemon prio=10 tid=0x00007efd20143000 nid=0x5b0 in Object.wait() [0x00007efd0d554000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d96dcd8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:561)
    - locked <0x00007efd4d96dcd8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:662)

"httpWorkerThread-4848-48" daemon prio=10 tid=0x00007efd18a82800 nid=0x5aa in Object.wait() [0x00007efd0db5a000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d09a620> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
    at java.lang.Object.wait(Object.java:485)
    at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
    - locked <0x00007efd4d09a620> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:120)


"GrizzlyPipelineStatWorkerThread-4848-2" daemon prio=10 tid=0x00007efd20033000 nid=0x579 waiting on condition [0x00007efd10c8b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d098f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"GrizzlyPipelineStatWorkerThread-4848-1" daemon prio=10 tid=0x00007efd2124f000 nid=0x578 waiting on condition [0x00007efd10d8c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d098f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"httpSSLWorkerThread-8181-48" daemon prio=10 tid=0x0000000042041800 nid=0x575 in Object.wait() [0x00007efd1108f000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d0c8308> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
    at java.lang.Object.wait(Object.java:485)
    at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
    - locked <0x00007efd4d0c8308> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
    at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)


"SelectorThread-4848" prio=10 tid=0x00007efd207ba800 nid=0x546 runnable [0x00007efd13fbe000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd4d9c24c0> (a sun.nio.ch.Util$1)
    - locked <0x00007efd4d9c24a8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd4d9c1d58> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
    - locked <0x00007efd4d9c27c0> (a [Ljava.lang.Object;)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)


"httpSSLWorkerThread-8080-49" daemon prio=10 tid=0x00007efd20bfc800 nid=0x541 in Object.wait() [0x00007efd144c3000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d0f97a8> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
    at java.lang.Object.wait(Object.java:485)
    at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
    - locked <0x00007efd4d0f97a8> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
    at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)


"SelectorThread-8181" prio=10 tid=0x00007efd2161c000 nid=0x50e runnable [0x00007efd177f6000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd4d3a6800> (a sun.nio.ch.Util$1)
    - locked <0x00007efd4d3a67e8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd4d3a6098> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
    - locked <0x00007efd4d3a6b00> (a [Ljava.lang.Object;)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)

"GrizzlyPipelineStatWorkerThread-8080-1" daemon prio=10 tid=0x00007efd20f4c800 nid=0x50a waiting on condition [0x00007efd17bfa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d120680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"GrizzlyPipelineStatWorkerThread-8080-0" daemon prio=10 tid=0x00007efd20966800 nid=0x509 waiting on condition [0x00007efd17cfb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d120680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"SelectorThread-8080" prio=10 tid=0x00007efd201a0800 nid=0x508 runnable [0x00007efd17dfc000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd4dc827a0> (a sun.nio.ch.Util$1)
    - locked <0x00007efd4dc82788> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd4dc82088> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
    - locked <0x00007efd4dc82aa0> (a [Ljava.lang.Object;)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv]]" daemon prio=10 tid=0x00007efd207af800 nid=0x503 waiting on condition [0x00007efd1c76c000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
    at java.lang.Thread.run(Thread.java:662)


"Timer-13" daemon prio=10 tid=0x00007efd202d2800 nid=0x4fe in Object.wait() [0x00007efd1c96e000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4ab21998> (a java.util.TaskQueue)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x00007efd4ab21998> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"Timer-9" daemon prio=10 tid=0x00007efd209ad000 nid=0x4f9 in Object.wait() [0x00007efd1d7d1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4a82c750> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x00007efd4a82c750> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"Thread-11" daemon prio=10 tid=0x00007efd183f6800 nid=0x4c0 waiting on condition [0x00007efd1efe7000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd48273738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at com.sun.enterprise.management.support.LoaderRegThread.processFOREVER(LoaderRegThread.java:263)
    at com.sun.enterprise.management.support.LoaderRegThread.run(LoaderRegThread.java:311)


"SelectorThread" daemon prio=10 tid=0x00007efd183ff800 nid=0x4ba runnable [0x00007efd1f5ed000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd48171e70> (a sun.nio.ch.Util$1)
    - locked <0x00007efd48171e58> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd48171a98> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:283)

I'm looking for general information about interpreting the dump so that I can diagnose problems in the future. Thanks.

Community
  • 1
  • 1
Allan Lykke Christensen
  • 1,337
  • 2
  • 14
  • 21
  • You can try to check memory usage first, perhaps you have GC taking more and more time before running out of memory. – axtavt Mar 22 '11 at 10:12
  • The memory usage seems to be quite stable. I will try to monitoring around the time when the application hangs. My app uses an XMLRPC client (inside EJB), Apache Solr integration (inside EJB via Solr4j) and provides file uploads (inside WAR using RichFaces), that I suspect may have resource leaks. – Allan Lykke Christensen Mar 22 '11 at 10:34
  • Are you looking out for any tools to help you diagnose Thread issues? – Santosh Gokak Mar 01 '12 at 14:50
  • You have a couple of threads RUNNABLE in sun.nio.ch.EPollArrayWrapper.epollWait(). You may be suffering from this JVM bug: http://bugs.sun.com/view_bug.do?bug_id=6403933 – Eduard Wirch Jan 29 '13 at 08:13

2 Answers2

10

Couple of things that might help you

How to move forward with current problem?

  • Based on this dump I can tell you there are couple of objects that are locked and waited on from multiple threads. Look for strings '- waiting on' and '- locked'. I don't know the logic of your app to tell much more.
toomasr
  • 4,731
  • 2
  • 33
  • 36
2

From your thread dump:

java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

looks like you have the well known BUG in JDK with the Selector spinning.

danidacila
  • 147
  • 1
  • 3
  • 2
    Could you expand a bit on that? Do you have a reference to the bug? – Allan Lykke Christensen Mar 01 '12 at 20:38
  • https://issues.apache.org/bugzilla/show_bug.cgi?id=52858 a colleague submitted this issue with tomcat and they fixed it. There are similar for jetty and other containers. It is described as a JDK bug, but all containers implement some workaround to get the Selector out of spinning. If you search for Jetty and the epollWait I think you'll find the most explanations. – danidacila Apr 03 '12 at 09:40