3
280K    ./saved_caches
112G    ./data
215G    ./commitlog
326G    .

How can this be explained? Is there a bug in cassandra 2.0.6?

kostas.kougios
  • 945
  • 10
  • 21
  • That looks quite bad as there should be an internal mechanism keeping the size of that directory around 4gb. Could you please give more details of your setup? What kind of workload are you running, are there any errors in your logs, what driver are you using ect ... – RussS Apr 15 '14 at 00:34
  • I am using the latest java datastax drivers. My workload is heavy insert based with small batches of 5-6 inserts at a time. I had a pseudo-cluster of 4 cassandras which are running on the same server and different disks. I had to decommision one of those (could that create the issue?) and I end up with 3 nodes in the cluster. As said, heavy data loading in the volumes of 2000 batches per second, sometimes (rarelly) cassandra dropped some mutations but other than that there is nothing weird in the log files. – kostas.kougios Apr 15 '14 at 09:51
  • What is your memtable threshold? Am amused to see a commitlog size greater than memtable threshold . – Ananth Apr 15 '14 at 13:55
  • do you mean memtable_total_space_in_mb? 2048 MB – kostas.kougios Apr 15 '14 at 19:30
  • is there a way for me to find which commit log files are in use and delete the rest? or a way to tell cassandra to flush the log files to sstables and delete them? – kostas.kougios Apr 15 '14 at 19:38
  • Related to [How to prevent Cassandra commit logs filling up disk space](http://stackoverflow.com/questions/31733395/how-to-prevent-cassandra-commit-logs-filling-up-disk-space). – Raedwald Jan 13 '16 at 11:37

1 Answers1

0

It seems I accidentally had disabled commitlog_total_space_in_mb. I've reenabled it and restarted cassandra. It took a long time due to the huge log files but it started and cleaned up the extra log files.

kostas.kougios
  • 945
  • 10
  • 21
  • How did you *disable* that value? If you do not provide a value, Cassandra chooses a default value: 8GB for a 64-bit JVM or 32MB for a 32-bot JVM. – Raedwald Jan 13 '16 at 11:01