What is the reference in "coldfusion.runtime.RWLock" on a stack trace

Question

In the following statement:

java.lang.Object.wait(Object.java:???)[Native Method]
- waiting on <0x6ab0eb48> (a coldfusion.runtime.RWLock)
coldfusion.runtime.RWLock.waitForLock(RWLock.java:154)

What does the "0x6ab0eb48" refer to? Is it a memory address on the heap? Is it some sort of reference to the stack? Is it possible to turn that reference into something useful like a line of code that is causing the blocking?

Full stack trace of same problem that occurred at a different time.

at java.lang.Object.wait(Native Method) - waiting on [0x00000000e0eac670] (a coldfusion.runtime.RWLock) at coldfusion.runtime.RWLock.waitForLock(RWLock.java:154) at coldfusion.runtime.RWLock.requestWriteLock(RWLock.java:124) - locked [0x00000000e0eac670] (a coldfusion.runtime.RWLock) at coldfusion.runtime.RWLock.requestLock(RWLock.java:46) at coldfusion.runtime.LockManager.requestNamedLock(LockManager.java:73) at coldfusion.tagext.lang.LockTag.doStartTag(LockTag.java:186) at cfmodules2eJOModule$CD35DE93ADBE111EADECF17DDC3D9E362etransfer173107414$funcSETMODULECONFIGSMEMENTO.runFunction(D:\JO3\jobOffice\config\transfer\definitions\modules.JOModule$CD35DE93ADBE111EADECF17DDC3D9E36.transfer:193) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:463) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:453) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:320) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2222) at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:362) at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2662) at cfMethodInvoker2ecfc210361352$funcINVOKEMETHOD.runFunction(D:\JO3\jobOffice\transfer\com\dynamic\MethodInvoker.cfc:47) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272) at cfTransferPopulator2ecfc918825853$funcPOPULATEONETOMANY.runFunction(D:\JO3\jobOffice\transfer\com\dynamic\TransferPopulator.cfc:113) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272) at cfDynamicManager2ecfc127705980$funcPOPULATEONETOMANY.runFunction(D:\JO3\jobOffice\transfer\com\dynamic\DynamicManager.cfc:93) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272) at cfTransfer2ecfc1432092572$funcLOADONETOMANY.runFunction(D:\JO3\jobOffice\transfer\com\Transfer.cfc:770) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272) at cfmodules2eJOModule$CD35DE93ADBE111EADECF17DDC3D9E362etransfer173107414$funcLOADMODULECONFIGS.runFunction(D:\JO3\jobOffice\config\transfer\definitions\modules.JOModule$CD35DE93ADBE111EADECF17DDC3D9E36.transfer:140) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2471) at cfmodules2eJOModule$CD35DE93ADBE111EADECF17DDC3D9E362etransfer173107414$funcGETMODULECONFIGSSTRUCT.runFunction(D:\JO3\jobOffice\config\transfer\definitions\modules.JOModule$CD35DE93ADBE111EADECF17DDC3D9E36.transfer:303) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272) at cfCPSecurity2ecfc397466374$funcPREPROCESS.runFunction(D:\JO3\jobOffice\sys\jo\core\m2plugin\CPSecurity.cfc:18) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272) at cfPluginManager2ecfc1495337580$funcPREPROCESS.runFunction(D:\JO3\jobOffice\MachII\framework\PluginManager.cfc:405) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272) at cfRequestHandler2ecfc1676430457$funcPROCESSEVENTS.runFunction(D:\JO3\jobOffice\MachII\framework\RequestHandler.cfc:275) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2471) at cfRequestHandler2ecfc1676430457$funcHANDLEREQUEST.runFunction(D:\JO3\jobOffice\MachII\framework\RequestHandler.cfc:208) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272) at cfmach2dii2ecfc1440816939$funcHANDLEREQUEST.runFunction(D:\JO3\jobOffice\MachII\mach-ii.cfc:210) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2471) at cfmach2dii2ecfc1440816939$funcONREQUESTSTART.runFunction(D:\JO3\jobOffice\MachII\mach-ii.cfc:121) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2471) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2290) at cfApplication2ecfc1530162842$funcONREQUESTSTART.runFunction(D:\JO3\wwwroot\Application.cfc:48) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308) at coldfusion.runtime.AppEventInvoker.invoke(AppEventInvoker.java:74) at coldfusion.runtime.AppEventInvoker.onRequestStart(AppEventInvoker.java:221) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:273) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:86) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:126) at coldfusion.CfmServlet.service(CfmServlet.java:175) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at com.intergral.fusionreactor.filter.FusionReactorFilter.b(FusionReactorFilter.java:386) at com.intergral.fusionreactor.filter.FusionReactorFilter.c(FusionReactorFilter.java:264) at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java:174) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

Is that the entire stack trace? If not, can you post it? From what I can find, it seems that that error tends to have something to do with using a named `cflock`. — Sean Walsh, Jun 15 '12 at 05:05

score 3 · Accepted Answer · edited May 23 '17 at 12:21

3

The hex reference does refer to an object on the Heap. the RWLock class is, as mentioned, likely to be related to a <cflock>. What you ideally need is a full stack trace. If you read down from the lock lines above, you'll get to references to your code, which include coldfusion line numbers, meaning you can then work out what's happening. If you're experiencing a deadlock, that reference may show up in other places in a full stack dump. To generate and analyse one of them, see this post for what works for me: When ColdFusion is maxing out the CPU, how do I find out what it's chewing/choking on?

edited May 23 '17 at 12:21

Community

1
1

answered Jun 15 '12 at 08:35

barnyr

5,678
21
28

Full stack trace now posted above. The offending line is in a TransferORM generated method. The method itself is unremarkable. It has a CfLock which is named with the class name and the PK of the entity. It had a couple of loops, but is not accessing any external resources that could cause the problem. What is confusing me is that the server gets into a state when I see around 10 threads with the same stack trace, all waiting on the same "address", eg, "0x6ab0eb48" these threads eventually complete, this keeps happening until we bounce the service. I want to see what code 0x6ab0eb48 is. – Dave Jun 17 '12 at 21:14
Until we bounce the service, the same problem keeps occurring (same lock, but different "address" every few minutes. I've done the "capture a stack trace every 20 seconds" method and use IBM DTA to see what is happening. The threads are moving (ie, executing), but slowly. I've used ProcExp.exe to view the CPU usage of each of the running threads, correlated to the stack traces in IBM TDA, and each one uses around 3-5% cpu. – Dave Jun 17 '12 at 21:18
to see what's living at an address, you should be able to use jmap: http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html - it'll write out the whole heap to disk and slow things down whilst its running so make sure you've got the space. You can then use jhat : http://docs.oracle.com/javase/6/docs/technotes/tools/share/jhat.html to review the heap dump. You'll need plenty of memory in order to run JHat (more than the size of the heap dump) – barnyr Jun 18 '12 at 10:36
1

My guess would be that something inside the RWLock (which is what the stack trace says it's waiting on) is not scoped as tightly as the lock name suggests or takes a long time to release. Can you temporarily edit the transfer-generated code with the cflock in it? I'd put a above the which might tell you what's going on when the slowdown happens – barnyr Jun 18 '12 at 10:40

What is the reference in "coldfusion.runtime.RWLock" on a stack trace

1 Answers1