How to get the on object monitor address when the java.lang.Thread.State in BLOCKED?

Question

We found a blocking issue for a application, and from the stack dump we found that many of the threads (one of which is catalina-exec-998 ) were blocked by thread catalina-exec-995 through the lock 0x000000065888b918, and we also found the blocker thread catalina-exec-995 is also in the state BLOCKED, it seems it tried to obtain the object monitor which was already owned by the other thread, however in the stack call there is no information for what object monitor address it tried to wait on,

so my question is how to find which object monitor\lock address it wait on?
Also BTW how to prevent object monitor from being JIT-compiled to make wait info always available by JVM option -XX:CompileCommand?

BTW, I also found a similar issue in java-thread-dump-waiting-on-object-monitor-what-is-it-waiting-on, and from the JVM Expert's answer I know the missing "waiting on " information maybe caused by the JIT compiler, and the solution in that URL is just suitable for Object.wait() in a synchronized object, however my situation is obviously different, as it tried to get object monitor instead of waiting for getting notification from the synchronized object.

The following information is the call stack for two threads mentioned about:

"catalina-exec-995" #1178 daemon prio=5 os_prio=0 tid=0x00007f6b7831f000 nid=0x15c0 waiting for monitor entry [0x00007f6b43ffc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.zip.ZipFile.getEntry(ZipFile.java:308)
- locked <**0x000000065888b918**> (a java.util.jar.JarFile)
at java.util.jar.JarFile.getEntry(JarFile.java:240)
at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:94)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:260)
at org.apache.catalina.webresources.StandardRoot.getResourcesInternal(StandardRoot.java:327)
at org.apache.catalina.webresources.CachedResource.validateResources(CachedResource.java:124)
- locked <0x000000068aa3c7a0> (a org.apache.catalina.webresources.CachedResource)
at org.apache.catalina.webresources.Cache.getResources(Cache.java:147)
at org.apache.catalina.webresources.StandardRoot.getResources(StandardRoot.java:315)
at org.apache.catalina.webresources.StandardRoot.getClassLoaderResources(StandardRoot.java:231)
at org.apache.catalina.loader.WebappClassLoaderBase.findResources(WebappClassLoaderBase.java:939)
at java.lang.ClassLoader.getResources(ClassLoader.java:1142)
at com.sun.jersey.spi.service.ServiceFinder.getResources(ServiceFinder.java:352)
at com.sun.jersey.spi.service.ServiceFinder.access$200(ServiceFinder.java:159)
at com.sun.jersey.spi.service.ServiceFinder$AbstractLazyIterator.setConfigs(ServiceFinder.java:720)
at com.sun.jersey.spi.service.ServiceFinder$AbstractLazyIterator.hasNext(ServiceFinder.java:732)
at com.sun.jersey.spi.service.ServiceFinder.toClassArray(ServiceFinder.java:595)
at com.sun.jersey.core.spi.component.ProviderServices.getServiceClasses(ProviderServices.java:318)
at com.sun.jersey.core.spi.component.ProviderServices.getServiceClasses(ProviderServices.java:311)
at com.sun.jersey.core.spi.component.ProviderServices.getServices(ProviderServices.java:159)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.initWriters(MessageBodyFactory.java:207)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:163)
at com.sun.jersey.api.client.Client.init(Client.java:342)
at com.sun.jersey.api.client.Client.access$000(Client.java:118)
at com.sun.jersey.api.client.Client$1.f(Client.java:191)
at com.sun.jersey.api.client.Client$1.f(Client.java:187)
at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
at com.sun.jersey.api.client.Client.<init>(Client.java:187)
at com.sun.jersey.api.client.Client.<init>(Client.java:170)
at com.sun.jersey.api.client.Client.create(Client.java:679)
at com.meritdata.tempo.foundation.security.util.RestfulUtils.request(RestfulUtils.java:33)
at com.meritdata.tempo.foundation.security.util.RestfulUtils.requestGet(RestfulUtils.java:28)
at com.meritdata.tempo.foundation.security.util.PaWebUtils.validatePaToken(PaWebUtils.java:344)
at com.meritdata.tempo.foundation.security.sso.PaSSO.readUserToken(PaSSO.java:137)
at com.meritdata.tempo.foundation.security.filter.UserLoginFilter.validatePortalLogin(UserLoginFilter.java:190)
at com.meritdata.tempo.foundation.security.filter.UserLoginFilter.onAccessDenied(UserLoginFilter.java:248)
at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)
at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)
at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)
at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.meritdata.tempo.foundation.security.filter.XssFilter.doFilter(XssFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.meritdata.tempo.foundation.security.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:152)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x0000000612c3a940> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000062d3e9728> (a java.util.concurrent.ThreadPoolExecutor$Worker)


"catalina-exec-998" #1181 daemon prio=5 os_prio=0 tid=0x00007f6b7c359800 nid=0x15c3 waiting for monitor entry [0x00007f6b43cf9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.zip.ZipFile.getEntry(ZipFile.java:308)
- waiting to lock <**0x000000065888b918**> (a java.util.jar.JarFile)
at java.util.jar.JarFile.getEntry(JarFile.java:240)
at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:94)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:260)
at org.apache.catalina.webresources.StandardRoot.getResourcesInternal(StandardRoot.java:327)
at org.apache.catalina.webresources.CachedResource.validateResources(CachedResource.java:124)
- locked <0x0000000691ae3b90> (a org.apache.catalina.webresources.CachedResource)
at org.apache.catalina.webresources.Cache.getResources(Cache.java:147)
at org.apache.catalina.webresources.StandardRoot.getResources(StandardRoot.java:315)
at org.apache.catalina.webresources.StandardRoot.getClassLoaderResources(StandardRoot.java:231)
at org.apache.catalina.loader.WebappClassLoaderBase.findResources(WebappClassLoaderBase.java:939)
at java.lang.ClassLoader.getResources(ClassLoader.java:1142)
at com.sun.jersey.spi.service.ServiceFinder.getResources(ServiceFinder.java:352)
at com.sun.jersey.spi.service.ServiceFinder.access$200(ServiceFinder.java:159)
at com.sun.jersey.spi.service.ServiceFinder$AbstractLazyIterator.setConfigs(ServiceFinder.java:720)
at com.sun.jersey.spi.service.ServiceFinder$AbstractLazyIterator.hasNext(ServiceFinder.java:732)
at com.sun.jersey.spi.service.ServiceFinder.toClassArray(ServiceFinder.java:595)
at com.sun.jersey.core.spi.component.ProviderServices.getServiceClasses(ProviderServices.java:318)
at com.sun.jersey.core.spi.component.ProviderServices.getServiceClasses(ProviderServices.java:311)
at com.sun.jersey.core.spi.component.ProviderServices.getServices(ProviderServices.java:159)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.initWriters(MessageBodyFactory.java:207)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:163)
at com.sun.jersey.api.client.Client.init(Client.java:342)
at com.sun.jersey.api.client.Client.access$000(Client.java:118)
at com.sun.jersey.api.client.Client$1.f(Client.java:191)
at com.sun.jersey.api.client.Client$1.f(Client.java:187)
at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
at com.sun.jersey.api.client.Client.<init>(Client.java:187)
at com.sun.jersey.api.client.Client.<init>(Client.java:170)
at com.sun.jersey.api.client.Client.create(Client.java:679)
at com.meritdata.tempo.foundation.security.util.RestfulUtils.request(RestfulUtils.java:33)
at com.meritdata.tempo.foundation.security.util.RestfulUtils.requestGet(RestfulUtils.java:28)
at com.meritdata.tempo.foundation.security.util.PaWebUtils.validatePaToken(PaWebUtils.java:344)
at com.meritdata.tempo.foundation.security.filter.UserLoginFilter.keepPaTokenInCookie(UserLoginFilter.java:496)
at com.meritdata.tempo.foundation.security.filter.UserLoginFilter.isAccessAllowed(UserLoginFilter.java:448)
at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)
at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)
at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.meritdata.tempo.foundation.security.filter.XssFilter.doFilter(XssFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.meritdata.tempo.foundation.security.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:152)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x0000000612c9dee0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000062ca8ffc8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Answer 1

In your case catalina-exec-995 and catalina-exec-998 threads compete for the same object monitor - an instance of JarFile with the address 0x000000065888b918. The stacktrace already has all the information.

You might have been confused by the fact that both threads are in BLOCKED state while one of them is waiting to lock the monitor, and the other has locked this monitor. This is absolutely normal situation: acquiring object monitor and switching thread state does not happen atomically.

A thread that is waiting to enter an object monitor first aquires the lock and then swiches from BLOCKED to RUNNABLE state. When you take a thread dump in between you may catch exactly the situation you see.

Let me demonstrate this with a simple example.

public class Sync implements Runnable {
    int n;

    public static void main(String[] args) {
        Sync sync = new Sync();
        new Thread(sync, "Sync1").start();
        new Thread(sync, "Sync2").start();
        new Thread(sync, "Sync3").start();
    }

    public void run() {
        for (;;) {
            inc();
        }
    }

    public synchronized void inc() {
        n++;
    }
}

A thread dump may look like

"Sync3" #13 prio=5 os_prio=0 tid=0x000000001deab800 nid=0x25d4 runnable [0x000000001e98f000]
   java.lang.Thread.State: RUNNABLE
    at Sync.run(Sync.java:13)
    at java.lang.Thread.run(Thread.java:748)

"Sync2" #12 prio=5 os_prio=0 tid=0x000000001deab000 nid=0x230c waiting for monitor entry [0x000000001e88f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at Sync.inc(Sync.java:18)
    - locked <0x000000076b8a1108> (a Sync)
    at Sync.run(Sync.java:13)
    at java.lang.Thread.run(Thread.java:748)

"Sync1" #11 prio=5 os_prio=0 tid=0x000000001deaa000 nid=0x16d4 waiting for monitor entry [0x000000001e78f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at Sync.inc(Sync.java:18)
    - waiting to lock <0x000000076b8a1108> (a Sync)
    at Sync.run(Sync.java:13)
    at java.lang.Thread.run(Thread.java:748)

Obviously, there is just one object monitor - the single instance of Sync. Both Sync1 and Sync2 threads are BLOCKED, but one of them is waiting to lock, and the other has locked the monitor.