OpsCenter rollups* tables are growing forever

Question

We are using DSE 4.8.8 with OpsCenter 5.2.4. All our OpsCenter.rollups* tables are growing forever, especially the rollups60 one:

Size of rollups60, 1 month

The keyspace OpsCenter uses org.apache.cassandra.locator.NetworkTopologyStrategy, DC1 1, DC2 1

Table settings:

CREATE TABLE "OpsCenter".rollups60 (
    key text,
    "timestamp" varint,
    value blob,
    PRIMARY KEY (key, "timestamp")
) WITH COMPACT STORAGE
    AND CLUSTERING ORDER BY ("timestamp" ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = '{"info": "OpsCenter management data.", "version": [5, 2, 1]}'
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 0
    AND gc_grace_seconds = 0
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.25
    AND speculative_retry = 'NONE';

The entries have no TTL on the columns key and timestamp and a TTL of 604618 on the column value.

Any idea how to fix this? I already tried a truncate as describes here: Datastax support

score 2 · Answer 1 · edited Nov 28 '18 at 13:42

You can set the 1 minute (rollup 60 table) ttl via the cluster config file, eg.: https://docs.datastax.com/en/opscenter/6.5/opsc/configure/opscChangingPerformanceDataExpiration_t.html

[cassandra_metrics]

1min_ttl = 43200
5min_ttl = 1209600
2hr_ttl = 0
24hr_ttl = 0

This can also be set in the address.yaml on each of your agents. Easier to set in OpsCenter cluster config once though.

You can tweak the table settings a little to improve the purging of tombstones.

ALTER TABLE "OpsCenter".rollups60 WITH
  compaction = {'class': 'SizeTieredCompactionStrategy',
    'tombstone_compaction_interval': '1',
    'unchecked_tombstone_compaction': 'true',
    'tombstone_threshold': '0.05'} AND 
  gc_grace_seconds = 0

To be more aggressive you can use LCS but this increases IO usage. For the sake of your scenario though its probably Ok as (at least with ttl option above) you wont actually have that much data to store.

ALTER TABLE "OpsCenter".rollups60 WITH
  compaction = {'class': 'LeveledCompactionStrategy', 
    'sstable_size_in_mb': '256mb',
    'tombstone_compaction_interval': '1',
    'unchecked_tombstone_compaction': 'true',
    'tombstone_threshold': '0.05'} AND 
  gc_grace_seconds = 0

OpsCenter rollups* tables are growing forever

1 Answers1