cassandra-4.0-beta4 Exception encountered during startup Requested permits (0) must be positive

Question

OS：Ubuntu20.04
JDK：JDK11
Cassandra version: apache-cassandra-4.0-beta4

Error message:

ERROR [main] 2021-03-25 14:16:15,561 CassandraDaemon.java:822 - Exception encountered during startup
org.apache.cassandra.exceptions.TruncateException: Error during truncate: java.lang.IllegalArgumentException: Requested permits (0) must be positive
    at org.apache.cassandra.cql3.statements.TruncateStatement.executeLocally(TruncateStatement.java:96)
    at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:323)
    at org.apache.cassandra.db.SystemKeyspace.clearAllEstimates(SystemKeyspace.java:1337)
    at org.apache.cassandra.service.StorageService.cleanupSizeEstimates(StorageService.java:3908)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:676)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:800)
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Requested permits (0) must be positive
    at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2329)
    at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2276)
    at org.apache.cassandra.db.ColumnFamilyStore.truncateBlocking(ColumnFamilyStore.java:2256)
    at org.apache.cassandra.cql3.statements.TruncateStatement.executeLocally(TruncateStatement.java:92)
    ... 6 common frames omitted
Caused by: java.lang.IllegalArgumentException: Requested permits (0) must be positive
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:189)
    at com.google.common.util.concurrent.RateLimiter.checkPermits(RateLimiter.java:430)
    at com.google.common.util.concurrent.RateLimiter.reserve(RateLimiter.java:285)
    at com.google.common.util.concurrent.RateLimiter.acquire(RateLimiter.java:273)
    at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1801)
    at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1981)
    at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1957)
    at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1945)
    at org.apache.cassandra.db.ColumnFamilyStore$4.run(ColumnFamilyStore.java:2242)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2325)
    ... 9 common frames omitted

score 2 · Answer 1 · answered Mar 25 '21 at 13:56

Question was also asked on the #cassandra channel on ASF Slack so I'm re-posting here for posterity.

Based on the stack trace, it looks like it's failing when trying to access the snapshot subdirectories of the system.size_estimates and system.table_estimates.

It's possible that the directories got corrupted as a result of the power outage (the reason for restarting the node in the first place) although I don't have sufficient information to confirm this.

In any case, the workaround is to manually delete the snapshot directories for the size_estimates/ and table_estimates/ in data/system/*. This will allow Cassandra to start successfully. Cheers!

P.S. 贾博超 confirmed on ASF Slack that the workaround allowed them to start C* on the node.

Thanks Erick! I must have missed this one on the mailing list. — Aaron, Mar 25 '21 at 14:22

score 1 · Answer 2 · answered Jul 23 '21 at 07:08

1

i found another solution

rm -rf apache-cassandra-4.0-beta4/data/saved_caches/*
rm -rf apache-cassandra-4.0-beta4/data/data/system/*

answered Jul 23 '21 at 07:08

Jia Bochao

31
2

Aaron · Answer 3 · 2021-03-25T14:26:36.507

Is this a restart of a failed node? It looks like something is trying to replay a TRUNCATE command, but something required isn't being sent along (as per CASSANDRA-14905).

To get around this, I would remove the system_schema keyspace from the disk and restart. It should re-stream the schema from another node.

If that doesn't work, you might end up having to wipe the data and rebuild the node. And if that still doesn't work, you may have to remove/rejoin it to the cluster.

Edit:

As per Erick's answer, it looks like the problem was with snapshots directories for the size_estimates and table_esimates tables in the system keyspace. Remove those, and you should be good.

score 0 · Answer 4 · answered Jul 22 '22 at 09:37

Noticed the same problem with a docker container. When I checked the logs

docker logs 59d4e2890a34

it highlighted this issue. i.e.

INFO  [main] 2022-07-22 09:24:16,961 ColumnFamilyStore.java:2242 - Truncating system.size_estimates
INFO  [main] 2022-07-22 09:24:16,964 ColumnFamilyStore.java:2279 - Truncating system.size_estimates with truncatedAt=1658481856962
org.apache.cassandra.exceptions.TruncateException: Error during truncate: java.lang.IllegalArgumentException: Requested permits (0) must be positive

Using

docker container rm 59d4e2890a34

I removed the existing container and re-created it. The new version ran fine. I'm not sure what corrupted system.size_estimates.

cassandra-4.0-beta4 Exception encountered during startup Requested permits (0) must be positive

4 Answers4