0

I have scheduled incremental repair for everyday. But while the repair is going on, our monitoring system reports COMPACTIONEXECUTOR_PENDING tasks.

I am wondering, if I can introduce a check, to see, if compaction is not running, before I trigger repair.

I should be able to check if compaction is running by parsing output of nodetool netstats and compactionstats command output.

I will proceed with repair if both of the following checks passes:

  1. nodetool netstats output contains Not sending any streams.
  2. nodetool compactionstats output contains pending tasks: 0

But I want to get some expert opinion before I proceed.

Is my understanding correct? I don't want to get into situation, in which, these checks are failing always and repair process is not getting triggered at all.

Thanks.

Erick Ramirez
  • 13,964
  • 1
  • 18
  • 23
Manojkumar Khotele
  • 963
  • 11
  • 25

1 Answers1

0

Compaction is occurring regularly in Cassandra. So I'm a bit scared that only triggering repair when pending_compactions=0 will result in repair not running enough. But it depends on your traffic of course, e.g. if you have few writes you won't do many compactions. You should probably add a max wait time for pending_compactions=0 so that after a specified time if the condition is not true repair will run anyway.

To answer your question. Nodetool uses JMX to fetch MBeans in Cassandra. You can see all available MBeans here: http://cassandra.apache.org/doc/latest/operating/metrics.html You want this MBean: 

org.apache.cassandra.metrics:type=Compaction name=PendingTasks

You can create your own JMX Client like this: How to connect to a java program on localhost jvm using JMX?

Or you can use jmxterm: https://github.com/jiaqi/jmxterm

My understanding is you could use it like this:

java -jar jmxterm-1.0.0-uber.jar
get -b org.apache.cassandra.metrics:type=Compaction name=PendingTasks
Simon Fontana Oscarsson
  • 2,114
  • 1
  • 17
  • 20