9

Absolutey stumped on this.

I have two controller integration tests that pass successfully. However, when running in Intellij or via gradle check, the JVM never exits. If I comment out the entire integration tests, the JVM exits cleanly.

When debugging any of the integration tests, I can hit pause and see that there are several threads in different states: WAITING, RUNNING, SLEEPING.

The database used in application.yml is purely an in-memory one:

 url: jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE

Changing this to file based does not fix the problem. Changing DB_CLOSE_ON_EXIT=TRUE does not help either.

I've tried removing @Rollback and even using @Transactional with a timeout, but that doesn't fix it.

Creating an integration test on a fresh project works with no deadlock/hanging/waiting.

I have moved back through revisions to find the changeset where this behaviour started, but the changes were purely in GSPs, Controllers and an additional assertion & test method in one of the integration tests.

The last lines in the logs are:

INFO org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext - Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@73386d72: startup date [Mon May 30 18:48:25 BST 2016]; root of context hierarchy
INFO org.springframework.context.support.DefaultLifecycleProcessor - Stopping beans in phase -2147483648
INFO org.grails.plugins.datasource.TomcatJDBCPoolMBeanExporter - Unregistering JMX-exposed beans on shutdown
INFO org.grails.plugins.datasource.TomcatJDBCPoolMBeanExporter - Unregistering JMX-exposed beans
INFO org.hibernate.tool.hbm2ddl.SchemaExport - HHH000227: Running hbm2ddl schema export
INFO org.hibernate.tool.hbm2ddl.SchemaExport - HHH000230: Schema export complete

I've tried cutting the integration test methods down to one method and the issue still occurs.

The versions I'm using are:

$ ~/apps/grails-3.1.5/bin/grails --version
|Grails Version: 3.1.5
|Groovy Version: 2.4.6
|JVM Version: 1.8.0_92

Windows 10 64bit.

Here's a thread dump.

I have no idea how to debug this further. Any ideas?

Alex
  • 8,093
  • 6
  • 49
  • 79
  • 1
    Did you try with to set `DB_CLOSE_ON_EXIT=TRUE` on the database url ? – Joch May 30 '16 at 19:04
  • Just tried it, doesn't fix it. Thanks though. Post updated. – Alex May 30 '16 at 20:30
  • 1
    Don't you have any thread that you didn't stop or set as daemon ? Did the JVM stop when you run integrations tests on a fresh project ? – Joch May 31 '16 at 17:10
  • I have no threads that I've created myself. The JVM stops when I run on a fresh project. It ran on these integration tests fine but suddenly stopped working. I've tried different transaction manages too (Not `@Rollback` but with `@Transactional` and a timeout) but that doesn't help. Totally clueless as to what's causing it. Going to debug with JMX next. – Alex May 31 '16 at 17:48
  • Thread dump here: https://gist.github.com/atc-/e32d6e2211032b21dc50f74037689fa5 – Alex Jul 03 '16 at 16:40
  • If you are facing this issue with `gradle check` and in `IntelliJ`, then I would recommend that you remove the `.gradle/caches/modules-2/files-2.1/*` and do a `gradle clean` then `gradle compile` and then run the test cases. – Nikhil Sharma Jul 05 '16 at 19:13
  • I will try this, but it occurs on CI too which is built inside a new Docker container each run. – Alex Jul 06 '16 at 08:28
  • If this is working when you create a new project, then the cause may be this. If doing the above steps solves your problem, then you can repeat that for your container also. Your container will be having this directory as well. – Nikhil Sharma Jul 06 '16 at 13:04
  • The container checks the code out and starts a fresh gradle build -- deps are downloaded fresh so not sure why caches would be relevant? – Alex Jul 06 '16 at 13:13
  • So, this did not work for you locally ? Also, you might be just running the container right, not creating it new from image each time you perform some operation ? – Nikhil Sharma Jul 07 '16 at 06:45
  • No such directory exists. I have `.gradle/2.9/taskArtifacts` and `2.13/taskArtifacts` which both contain a few files (`fileHashes.bin`, `cache.properties` and its lock file) – Alex Jul 07 '16 at 19:30
  • You won't find that in your project directory. Your home directory will be having that. For windows, you should find it in your `GRADLE_HOME` path I think. Also, only the current version you are using should be present in your project root. Check your gradleWrapperVersion in `gradle.properties` or simply delete your `.gradle` folder in the project root and run `grails clean`, this will fix that. – Nikhil Sharma Jul 08 '16 at 05:58
  • Thanks. I did this (Windows 10, in git bash): `rm .gradle gradle gradlew gradlew.bat -rf`, `rm /c/Users/atc/.gradle/caches/modules-2/files-2.1 -rf`, `rm .gradle gradle gradlew gradlew.bat -rf`, `~/apps/gradle-2.13/bin/gradle wrapper`, `./gradlew clean`, `grails clean`and finally `grails test-app` but I end up with hibernate exceptions. Baaah! – Alex Jul 09 '16 at 10:29
  • Still freezes in IntelliJ (after all integration tests pass) – Alex Jul 09 '16 at 12:26
  • Looks like your test cases are getting executed in forked mode and hence somehow a thread is getting parked. you are running test cases using what command. Please edit your question to mention that too. And try running a single integration test case using grails test-app :integration testClass. – Vinay Prajapati Aug 31 '16 at 05:07
  • I've said in the question already: using `gradle check` and IntelliJ -- both behave incorrectly. – Alex Aug 31 '16 at 11:47

1 Answers1

1

I would turn on debug level logging. Also, if you can, I would upgrade grails to something post 3.1.9. (3.1.11 is current as I write this.)

Right around grails v3.1.5 there were configuration inconsistencies between Grails and Hibernate. The grails team was quickly upgrading several interfaces, and they got through it quickly.

The result was that you didn't end up running the configuration that you thought you were. It also affected cache and transaction management.

At the time, I had to create redundant configs to make sure Grails was getting configs at one level, hibernate at another. You don't have to do this anymore, but at the time, I had to use a config like the one listed here.

To find the problem, I turned on debug logging for grails and hibernate and all of my database drivers and waded all of the way through it.

This plugin also helps with the detailed monitoring info:

Community
  • 1
  • 1
fcnorman
  • 1,154
  • 9
  • 19