liquibase.lockservice.StandardLockService.waitForLock

Question

related: Issue on github: https://github.com/broadinstitute/cromwell/issues/7009

I'm using a bioinformatics java program named cromwell (https://github.com/broadinstitute/cromwell/) which, as far as I understand, uses liquibase and hsqldb to store data. I'm not familiar with liquibase. The hsqldb is a local file:

    jdbc:hsqldb:file:cromwell-executions/cromwell-db/cromwell-db;
    shutdown=false;
    hsqldb.default_table_type=cached;hsqldb.tx=mvcc;
    hsqldb.result_max_memory_rows=10000;
    hsqldb.large_data=true;
    hsqldb.applog=3;
    hsqldb.lob_compressed=true;
    hsqldb.script_format=3

Whenever I try to start the program I got the following stacktrace:

java  -Dconfig.file=${PWD}/app.conf -jar  ${CROMWELL_JAR} run test.wdl --inputs input.json
(...)
2023-02-08 16:32:11,54] [info] checkpointClose synched
[2023-02-08 16:32:11,57] [info] checkpointClose script done
[2023-02-08 16:32:11,57] [info] dataFileCache commit start
[2023-02-08 16:32:11,57] [info] dataFileCache commit end
[2023-02-08 16:32:11,69] [info] checkpointClose end
[2023-02-08 16:32:11,69] [info] Checkpoint end - txts: 5342
[2023-02-08 16:32:21,70] [info] Checkpoint start
[2023-02-08 16:32:21,70] [info] checkpointClose start
[2023-02-08 16:32:21,70] [info] checkpointClose synched
[2023-02-08 16:32:21,74] [info] checkpointClose script done
[2023-02-08 16:32:21,74] [info] dataFileCache commit start
[2023-02-08 16:32:21,76] [info] dataFileCache commit end
[2023-02-08 16:32:21,82] [info] checkpointClose end
[2023-02-08 16:32:21,82] [info] Checkpoint end - txts: 5348
[2023-02-08 16:32:21,89] [error] Failed to instantiate Cromwell System. Shutting down Cromwell.
liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fdb0:cafe:d0d0:ceb4:ba59:9fff:fec3:33de%p1p1 (fdb0:cafe:d0d0:ceb4:ba59:9fff:fec3:33de%p1p1) since 2/8/23, 4:23 PM
    at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:270)
    at liquibase.Liquibase.lambda$update$1(Liquibase.java:214)
    at liquibase.Scope.lambda$child$0(Scope.java:180)
    at liquibase.Scope.child(Scope.java:189)
    at liquibase.Scope.child(Scope.java:179)
    at liquibase.Scope.child(Scope.java:158)
    at liquibase.Liquibase.runInScope(Liquibase.java:2405)
    at liquibase.Liquibase.update(Liquibase.java:211)
    at liquibase.Liquibase.update(Liquibase.java:197)
    at cromwell.database.migration.liquibase.LiquibaseUtils$.updateSchema(LiquibaseUtils.scala:74)
    at cromwell.database.migration.liquibase.LiquibaseUtils$.updateSchema(LiquibaseUtils.scala:46)
    at cromwell.services.ServicesStore$EnhancedSqlDatabase$.$anonfun$initialized$1(ServicesStore.scala:11)
    at cromwell.services.ServicesStore$EnhancedSqlDatabase$.$anonfun$initialized$1$adapted(ServicesStore.scala:11)
    at cromwell.database.slick.SlickDatabase.$anonfun$withConnection$1(SlickDatabase.scala:156)
    at slick.jdbc.SimpleJdbcAction.run(StreamingInvokerAction.scala:70)
    at slick.jdbc.SimpleJdbcAction.run(StreamingInvokerAction.scala:69)
    at slick.basic.BasicBackend$DatabaseDef$$anon$3.liftedTree1$1(BasicBackend.scala:276)
    at slick.basic.BasicBackend$DatabaseDef$$anon$3.run(BasicBackend.scala:276)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1589)

so ,as far as I understand liquibase is not able to lock a file although I deleted all the (+hidden) files in my working directory.

I don't care about db versions, is there a genetic way to disable liquibase (e.g using java properties ?)
is there a place (my home ?) where liquibase writes the locks ?
How can I fix this ?

Are you sure it is not just a permission error? Workflow pipelines often give notoriously misleading error messages, or bury the actual error message in a .err log in a special work or logging directory. — Pallie, Feb 09 '23 at 13:27
From this: https://stackoverflow.com/questions/15528795/liquibase-lock-reasons it looks like liquibase stores the locks in the database itself, in a table called `DATABASECHANGELOGLOCK` ... maybe — tim_yates, Feb 09 '23 at 13:57
@tim_yates yes I saw this, but here the database is embeded in cromwell, and I removed everything (files , directories, ...) before running the tool. — Pierre, Feb 09 '23 at 18:06

score 1 · Accepted Answer · answered Feb 09 '23 at 21:01

1

Cromwell developer here. This can happen if the server stopped unexpectedly partway through a previous migration and never released its lock.

The easiest solution is to delete the HSQL pages from disk and start over. Another option would be to use a standalone DB client program to unset the value in DATABASECHANGELOGLOCK.

answered Feb 09 '23 at 21:01

aednichols

2,292
2
19
28

Thanks for your answer. As a I said above, I deleted everything under the directory `cromwell-executions` which also contains the hsqldb database and I tried several times. So this is a recurrent error. Are some files stored elsewhere ? – Pierre Feb 10 '23 at 22:23
The other commenter is correct that the lock is stored in the DB. I don’t use HSQLDB in my work but from searching online it looks like you can change the connection string to `jdbc:hsqldb:file:~/new-db`. It will make a new file with a clean state; and if you run into trouble again, you know what to delete. Hope this helps! – aednichols Feb 11 '23 at 23:10
See abobe; my connection URL was `jdbc:hsqldb:file:cromwell-executions/cromwell-db/cromwell-db`. The whole directory "cromwell-executions" was always deleted before running/trying cromwell. – Pierre Feb 12 '23 at 15:56
1

I believe you, but can you still try my suggestion to use a new file path outside `executions`? – aednichols Feb 12 '23 at 20:08
Oh I see, I'll try next tuesday – Pierre Feb 12 '23 at 22:00

liquibase.lockservice.StandardLockService.waitForLock

1 Answers1