4

I am using a LogAppender class which extends a Console Appender and internally uses a Sendmail function to send mails if the log level is ERROR. (I know I could have used a SMPT appender for mailing, but this is how the infrastructure is so any please avoid any comments on this).

My application is split into different threads, and the issue mentioned below happends sporadically.(But some observation tells me that it might be due to one thread taking more time to initalize).

I get the following dump in jstack:

    "pool-68-thread-1":
  waiting to lock Monitor@0x0000000005496818 (Object@0xfffffd7e390328b0, a crazycode/api/core/common/log/LogAppender),
  which is held by "main"
"main":
  waiting to lock Monitor@0x00000000064dbfc0 (Object@0xfffffd7dfb1f8908, a org/apache/log4j/Logger),
  which is held by "pool-68-thread-1"

Found a total of 1 deadlock.

Thread t@328: (state = BLOCKED)
 - org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) @bci=13, line=202 (Interpreted frame)
 - org.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=14, line=388 (Interpreted frame)
 - org.apache.log4j.Category.log(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=34, line=853 (Compiled frame)
 - crazycode.team2Api.common.logging.DESLogger.debug(java.lang.Object) @bci=12, line=240 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.refreshCache(java.util.List, java.lang.Object, java.lang.String) @bci=33, line=229 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.initialize(java.lang.Object) @bci=70, line=285 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.init() @bci=18, line=110 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.initialize() @bci=1, line=98 (Interpreted frame)
 - crazycode.api.alert.handler.MemoHandler.initRealHandler() @bci=4, line=573 (Interpreted frame)
 - crazycode.api.alert.handler.RealAlertCoordinator.initHandler(crazycode.api.alert.handler.AlertHandler, int, int, boolean, boolean) @bci=61, line=47 (Interpreted frame)
 - crazycode.api.alert.handler.run.BaseAlertHandlerRun$1$1.run() @bci=72, line=229 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame)
 - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread t@327: (state = BLOCKED)
 - org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) @bci=13, line=202 (Interpreted frame)
 - org.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=14, line=388 (Interpreted frame)
 - org.apache.log4j.Category.log(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=34, line=853 (Compiled frame)
 - crazycode.team2Api.common.logging.DESLogger.debug(java.lang.Object) @bci=12, line=240 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.refreshCache(java.util.List, java.lang.Object, java.lang.String) @bci=33, line=229 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.initialize(java.lang.Object) @bci=70, line=285 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.init() @bci=18, line=110 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.initialize() @bci=1, line=98 (Interpreted frame)
 - crazycode.api.alert.handler.ResearchDocumentHandler.initRealHandler() @bci=86, line=957 (Interpreted frame)
 - crazycode.api.alert.handler.RealAlertCoordinator.initHandler(crazycode.api.alert.handler.AlertHandler, int, int, boolean, boolean) @bci=61, line=47 (Interpreted frame)
 - crazycode.api.alert.handler.run.BaseAlertHandlerRun$1$1.run() @bci=72, line=229 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame)
 - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread t@322: (state = BLOCKED)
 - org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) @bci=13, line=202 (Interpreted frame)
 - org.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=14, line=388 (Interpreted frame)
 - org.apache.log4j.Category.log(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=34, line=853 (Compiled frame)
 - crazycode.team2Api.common.logging.DESLogger.debug(java.lang.Object) @bci=12, line=240 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.refreshCache(java.util.List, java.lang.Object, java.lang.String) @bci=33, line=229 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.initialize(java.lang.Object) @bci=70, line=285 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.init() @bci=18, line=110 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.initialize() @bci=1, line=98 (Interpreted frame)
 - crazycode.api.alert.handler.RkdResearchDocumentHandler.initRealHandler() @bci=86, line=779 (Interpreted frame)
 - crazycode.api.alert.handler.RealAlertCoordinator.initHandler(crazycode.api.alert.handler.AlertHandler, int, int, boolean, boolean) @bci=61, line=47 (Interpreted frame)
 - crazycode.api.alert.handler.run.BaseAlertHandlerRun$1$1.run() @bci=72, line=229 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame)
 - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)

Can someone shed some light as to what could be the issue and what are the ways to solve it??

Neeraj
  • 8,408
  • 8
  • 41
  • 69
  • I can't say for sure, but this seems like a bug in your own code, not in log4j. Check all your synchronized blocks and synchronized methods, for main and for the threads in your pool. Look for main locking A (some object's monitor) and then B (another one), and a thread from the pool locking B and then A. – Eran Zimmerman Gonen Aug 23 '11 at 07:22

2 Answers2

7

I saw a similar trace today in my own code. Maybe it applies to you too?

Summary:

  • Before (deadlock): logger.info(x); (where x is a complicated object that acquires a read lock within toString())
  • After (ok): logger.info(x.toString());

Details:

In one thread I called logger.info(x), which called Category.callAppenders(LoggingEvent), which synchronizes on an appender and then calls x.toString(), which tried to acquire a read lock on x and blocked.

In another thread, I had a method that held the write lock on x while calling logger.info(""), which also called callAppenders, tried to synchronize on an appender, and blocked. Here's the relevant part of the trace:

Thread One [WAITING]
sun.misc.Unsafe.park(native method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
(... my ReadWriteLock$ReadLock.lock(Lock.java:256))
(... my Object's toString() ...)
org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
org.apache.log4j.xml.XMLLayout.format(XMLLayout.java:148)
org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
org.apache.log4j.Category.callAppenders(Category.java:206)
org.apache.log4j.Category.forcedLog(Category.java:391)
org.apache.log4j.Category.log(Category.java:856)
Log.ndebug(Log.java:965)



Thread Two [BLOCKED; waiting to lock org.apache.log4j.Logger@1623f]
org.apache.log4j.Category.callAppenders(Category.java:204)
org.apache.log4j.Category.forcedLog(Category.java:391)
org.apache.log4j.Category.log(Category.java:856)
Log.debug(Log.java:820)
(.. something that acquires my reentrant ReadWriteLock$WriteLock ...)
yonran
  • 18,156
  • 8
  • 72
  • 97
  • was this solved after making the changes you descire in the After label? – Neeraj Sep 13 '11 at 07:06
  • Yes. My code was causing a deadlock because logger.info(x) called x.toString() which acquired my own lock, and another thread that held my own lock called logger.info(""). – yonran Sep 13 '11 at 18:18
0

Are there stacks for the "main" thread and "pool-68-thread-1" threads? I'd look at the first line down the stack that's your code in each of those threads to figure out what's possibly causing the deadlock.

Marcus
  • 2,128
  • 20
  • 22
  • I'd also look into what adapters you're using. I've seen a few threads where people used custom adapters which caused issues. E.g., https://issues.apache.org/bugzilla/show_bug.cgi?id=41214 – Marcus Jul 11 '12 at 01:24