3

Recently I faced an issue in a customer setup with a 3 node cluster, where one node went down and came online only after 12 days. The default gc_grace_seconds for most of the table has been set to 1 day in our scenario and there are a lot of tables.

When this down node came up, stale data from this node got replicated to the other nodes leading to zombie data in all the three nodes.

A solution that I could think of was to clean the node before making it join the cluster and then run a repair which could prevent the occurrence of zombie data. Could there be any other possible solution to avoid this issue where I don't need to clean the node.

Vishal
  • 55
  • 3

1 Answers1

5

You should never bring a node back online if it has been down for longer than the shortest gc_grace_seconds.

This is a challenge in environments where GC grace is set to a very low value. In these situations, the procedure is to completely rebuild the node as if it was never part of the cluster:

  1. Completely wipe all contents of data/, commitlog/ and saved_caches/.
  2. Remove the node's IP from its seeds list if it is listed as a seed node.
  3. Replace the node with itself using the replace_address flag.

Cheers!

Erick Ramirez
  • 13,964
  • 1
  • 18
  • 23