9

I have a web app deployed to Tomcat 7.0.54 that uses a datasource to connect to an Oracle 11g database. The datasource is configured in META-INF/context.xml and I've placed ojdbc7.jar in <tomcat-install-dir>/lib. I use a JNDI lookup to retrieve the datasource which I store in a singleton so that each DAO class can use it.

Everything works as expected, however when I undeploy the application (via Tomcat manager app) I see in the logs:

Oct 03, 2014 3:06:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/myapp] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak.
Oct 03, 2014 3:06:57 PM org.apache.catalina.startup.HostConfig undeploy
INFO: Undeploying context [/myapp]

When I debug I can see this thread gets created as soon as the database is accessed (via the datasource).

My datasource config:

<Context antiResourceLocking="false">
    <Resource name="jdbc/myapp" auth="Container" 
        type="javax.sql.DataSource" driverClassName="oracle.jdbc.OracleDriver" 
        maxActive="20" maxIdle="10" maxWait="-1"
        username="myuser" password="mypass"
        url="jdbc:oracle:thin:@myserver:1521:mysid"
        removeAbandoned="true" removeAbandonedTimeout="10" logAbandoned="true"
        validationQuery="SELECT 1 FROM DUAL"   
        testOnBorrow="true" testOnReturn="true" testWhileIdle="true"  
        timeBetweenEvictionRunsMillis="1800000" numTestsPerEvictionRun="3"  
        minEvictableIdleTimeMillis="1800000"
    />
</Context>

EDIT

Further investigation has revealed that the issue occurs whether or not the datasource is accessed during application (or servlet) initialisation.

In actual fact, the problematic thread is only created, and thus the issue only exists, when using the 12c versions of Oracle's JDBC drivers (either ojdbc6.jar or ojdbc7.jar).

If I revert to using the 11.2.0.4 version of ojdbc6.jar the thread is never created and the memory leak warning never appears.

Should I downgrade JDBC driver (as suggested in https://stackoverflow.com/a/9177263/4105953)?

Community
  • 1
  • 1
finchie
  • 123
  • 1
  • 2
  • 8
  • Make sure that your oracle driver isn't also in the `WEB-INF/lib` directory. As that would auto register and start some of the Oracle JDBC stuff. See also [here](http://stackoverflow.com/questions/3320400/to-prevent-a-memory-leak-the-jdbc-driver-has-been-forcibly-unregistered). – M. Deinum Oct 07 '14 at 09:29
  • @M.Deinum thanks, but I've already ensured that the JDBC driver is only in `tomcat/lib` and not the application's `WEB-INF/lib` – finchie Oct 07 '14 at 11:08
  • Just in case someone else runs across this like I did; was using the ojdbc8 v12.2.0.1 which purportedly has this issue fixed. Was running it in intellij and couldn't start the server due to this message and subsequent crash. Solved by simply invalidating and restarting intellij. /shrug – buddyp450 Jun 10 '20 at 16:57

3 Answers3

1

I found a lengthy discussion about the subject here. The conclusion is that it creates a "fixed size memory leak", i.e. subsequent redeploys will not increase the memory leak.
I do not have Oracle Support access, but the bug ID mentioned in the discussion is 16841748 (May 2013, might be solved now).

A possible workaround is to actually use the datasource once (get connection, do dummy query, close connection) when Tomcat is started via a custom servlet that is configured to "load-on-startup" in tomcat/conf/web.xml. This should start the Oracle driver thread(s) (see also the FAQ about driver threads) outside of the scope of the class-loader of your web-app, thus preventing the "fixed size memory leak".

Note that a similar issue exists for the MySQL JDBC driver but has a decent solution. Such a solution may exist for a recent version of the Oracle JDBC driver (I don't know).

Community
  • 1
  • 1
vanOekel
  • 6,358
  • 1
  • 21
  • 56
  • Thanks for the "load-on-startup" workaround suggestion - unfortunately this has no effect and the memory leak warning persists upon undeploy. – finchie Oct 09 '14 at 15:12
  • Hmm, looked it up in the decompiled code and found that each oracle.jdbc.driver.PhysicalConnection creates a "BlockSource" thread. So that explains why "load-on-startup" does not work (I assumed it was a general thread started by the driver, not per connection). Using the older driver version, as you described in your updated question, looks like a better solution. – vanOekel Oct 10 '14 at 13:45
  • 2
    I think that discussion is actually about a similar but unrelated issue, which is why the workaround had no effect. There is an Oracle doc ID for this, though, and a patch available: [JDBC Thin Driver Leaks Threads With Tomcat (Doc ID 1901449.1)](https://support.oracle.com/epmos/faces/DocContentDisplay?id=1901449.1) – Mike Sep 30 '15 at 09:13
  • It is not a "fixed size" leak. It creates [classloader leaks](http://stackoverflow.com/a/17975255/324152) which will soon or later end up in an OutOfMemory error. When you look at the heap dump, you can clearly see that all stopped web application class loaders could not be garbage collected as their classloader is still referenced by `oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.contextClassLoader` – MRalwasser Sep 06 '16 at 15:43
0

This is normal, it happens when you hot deploy in Tomcat. It will not cause you any problems normally in production because you don't generally keep hot deploying updates in production, you just stop and restart the server instead.

Rocky Pulley
  • 22,531
  • 20
  • 68
  • 106
  • 5
    Other applications are deployed to our production Tomcat, some with much larger userbases than this app! Surely I shouldn't have to restart the production Tomcat just to redeploy an app? Isn't that what the Tomcat Manager app is for? – finchie Oct 03 '14 at 15:19
-1

This problem doesn't happen only with Oracle JDBC driver (oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser).

If any of the 3rd party dependency starts a daemon thread for carrying out some task, then while stopping the Tomcat logs Warning message for them also.

Also, this problem exists with Tomcat 8.5.X versions.

Solution: What i found as a solution is to get the Thread Group for the current thread and interrupt it. It will make sure that before shutting down the Tomcat all it's daemon threads are killed.

below code should be added to the "contextDestroyed" method ThreadGroup threadGroup = Thread.currentThread().getThreadGroup(); threadGroup.interrupt();

subrat patra
  • 47
  • 3