I have a suite of JUnit tests that works when run out of STS/Eclipse, but does not work when run from Bamboo using Jacoco. It used to work when we were running the tests out of Jenkins using Cobertura. When I say work, I mean the tests are erroring out, and it is specifically the tests that touch our SQL database (Oracle). The tests that don't touch the database do not have the error. We are also now on better/faster hardware than we used to be so that might be it. We are also running now Java 8 instead of 7. But like I said, the tests run fine from STS/Eclipse. We changed a LOT of things at once, so I'm at a bit of a loss as to what could be doing this.
The error is specifically "error occurred at recursive SQL level 1 ORA-01000 Maximum Open Cursors exceeded".
We use Springsource Framework, ojdbc6 and ucp, and the error occurs with following stack trace:
org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL []; SQL state [60000]; error code [604]; ORA-00604: error occurred at recursive SQL level 1 ORA-01000: maximum open cursors exceeded; nested exception is java.sql.SQLException: ORA-00604: error occurred at rcursive SLQ level 1 ORA-01000: maximum open cursors exceeded
at
org.springframework.jdbc.support.AbstractFallbackSLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at
org.springframework.jdbc.core.JdbcTemplate.execute (JdbcTemplate.java:660)
at
mycompany.db.StatementManager.execute(StatementManager.java:183)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at
com.sun.proxy.$Proxy27.execute(Unknown Source)
at
mycompany.db.SomeDbCode.clear(SomeDbCode.java:48)
at
........more AOP layers that are irrelevant ....
mycompany.db.TestSomeDb.setup(TestSomeDb.java:115) //this runs before each set of unit tests to truncate the tables to pristine state with a @Before
at
..... layers up to the runner ...
at
org.springframework.test.context.junit4.SpringJUnit4ClassRnner.run (SpringJUnit4ClassRunner.java:163)
Caused by: java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1
ORA-01000: maximum open cursors exceeded
at
oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:439)
at
oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:395)
at
oracle.jdbc.driver.T4C80all.processError(T4C80all.java:802)
at
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:436)
at
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
at
oracle.jdbc.driver.T4C80all.do0ALL(T4C80all.java:521)
at
oracle.jdbc.driver.T4CPreparedStatement.do0all8(T4CPreparedStatement.java:205)
at
oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1008)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1307)
at
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3449)
at
oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3530)
at
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1350)
at
oracle.ucp.jdbc.proxy.StatementProxyFactory.invoke(StatementProxyFactory.java:230)
at
oracle.ucp.jdbc.proxy.StatementProxyFactory.invoke(StatementProxyFactory.java:124)
at
com.sun.proxy.$Proxy113.executeUpdate(Unknown Source)
at
mycompany.db.StatementManager$ExecuteFiller.doInPreparedStatment(StatementManager.java:44)
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:644)
This by the way all from trying to call:
TRUNCATE TABLE sometablename
I think i've figured out the root cause though not yet the best way to fix it (open to ideas).
Each set of tests opens some cursors. As that set of tests finishes running, Spring starts cleaning up cursors. But then the next class of tests starts up and starts opening new cursors. I was watching the "open cursors" number do a growing sinusoidal by running queries How to find Current open Cursors in Oracle. Apparently my old hardware was so slow the cleanup would happen faster than the growth, but with the new stuff it eventually hits the limit of 300 (so at some point someone else on the team realized the default 50 is too few, though of course there is no documentation about how 300 was determined to be right), and any tests after that error out with that error.
Things I tried to mitigate:
Give all the tests that touch the database a superclass called TouchesDatabase with a @AfterClass method that sleeps for 10 seconds.
This actually fixed up all but one of my tests, but it also made the build take twice as long to build.
So I thought, ok, clearly some tests open more cursors than others, so instead of doing 10 sec, I'll query for how many cursors are open and if it's more than 100 I'll sleep another second. Unfortunately it seems the cursor query I'm using (may have to try a different one) is either overcounting things or doing something weird since it seems to take even longer to drop below 100 cursors than just a flat 10 sec sleep. When I had to leave for the day it was doing like the 3rd or 4th db test class (out of 30ish) and had already been running for 30 minutes, so i'll take a look in the morning, but obviously not ideal.
If I can't find a better query for cursors, then I'll start inspecting each test to see if there is a smarter way to do things there. We have not had any issues with the production application, btw, just the tests.
