0

I'm using Bamboo to build, test, and deploy commits to a server. However, every once in a while I get a failed deployment on the Tomcat end. Bamboo doesn't know this happened, so I have to run a manual build when the application doesn't start properly.

As far as I can tell, it's due to running Quartz tasks. Is there a way to safely deploy the WAR but wait until all jobs are done? Is better to build the plan to first undeploy before deploying?

I've pulled my Catalina logs to highlight the issue:

Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-5] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-6] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-8] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-10] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [Thread-212] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:15 PM org.apache.catalina.startup.HostConfig checkResources
INFO: Undeploying context [/emdb]
Sep 12, 2012 12:01:16 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [....webapps\emdb\WEB-INF\lib] could not be completely deleted. The presence of the remaining files may cause problems
Sep 12, 2012 12:01:16 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [....webapps\emdb\WEB-INF] could not be completely deleted. The presence of the remaining files may cause problems
Sep 12, 2012 12:01:16 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [....webapps\emdb] could not be completely deleted. The presence of the remaining files may cause problems
Sep 12, 2012 12:01:16 PM org.apache.catalina.startup.ExpandWar delete
SEVERE: [....webapps\emdb] could not be completely deleted. The presence of the remaining files may cause problems
Sep 12, 2012 12:01:17 PM org.apache.catalina.startup.HostConfig deployWAR
John Giotta
  • 16,432
  • 7
  • 52
  • 82

1 Answers1

1

You are right, Quartz scheduler is not stopping threads created to run jobs (worker thread pool). You need to call QuartzScheduler.shutdown(boolean) method when application shuts down. I'm surprised that Grails Quartz plugin is not doing it for you.

There are other reasons for this to happen. You might have some running jobs and Quartz hangs waiting for them to finish (even when proper shutdown was called). Tomcat is impatient and kills the whole thing in the middle. Check out (uncodumented?) org.quartz.scheduler.interruptJobsOnShutdown and org.quartz.scheduler.interruptJobsOnShutdownWithWait options.

Note that ShutdownHookPlugin won't help you since you are not shutting down the whole JVM, only undeploying one application.

Also the error message lists Thread-212 as one of the causes. This is most likely your custom thread (always name threads!) which also has to be interrupted.

See also

Community
  • 1
  • 1
Tomasz Nurkiewicz
  • 334,321
  • 69
  • 703
  • 674
  • How can I name my threads? I have 2 quartz jobs in grails, but I'm not aware of way to name them. – John Giotta Sep 24 '12 at 14:04
  • @JohnGiotta: Quartz names his threads, e.g. `quartzScheduler_Worker-N`. But I see `Thread-212` in your app. It was either created by you or by some library. Check out [`Thread.setName()`](http://docs.oracle.com/javase/7/docs/api/java/lang/Thread.html#setName(java.lang.String)). – Tomasz Nurkiewicz Sep 24 '12 at 14:29
  • Success! Thank you, sir. It took some extra digging, but tests have been successful for the past 15 deployments over the last 5 hours. – John Giotta Sep 24 '12 at 19:43
  • @JohnGiotta: glad to hear that. Can you explain further what was the problem and how you solved it? – Tomasz Nurkiewicz Sep 24 '12 at 20:06
  • 1
    The first problem was the Quartz config wasn't created, so I added what was missing. Then set `waitForJobsToCompleteOnShutdown` flag to true. – John Giotta Sep 24 '12 at 20:36