6

In order to remove a node (working / non working) which command is applicable in which situation?

Bhuvan Rawal
  • 386
  • 4
  • 15

3 Answers3

16

Decommission streams data from the leaving node. Therefore, you are guaranteed to maintain the same consistency that you had at the time you start the operation.

Removenode streams data from any available node that owns the range. It's possible you could violate consistency (and even potentially lose data) depending on your time since repair and the consistency level you use to write data.

Typically, you should prefer decommission if possible. Imagine you have 5 node cluster (A-E), and a given write would go to B, C, and D. You write with quorum, so for some reason C is down, but a write goes to B and D. When C comes back online, D needs to be removed from the cluster (you're downsizing, or changing drives, or something).

  • If D is online, you run decommission, and D sends its data to E - you keep 2 replicas of all of the data, and you'll be able to run repair later to get the write to C.

  • If D is offline, or if you run removenode instead of decommission, you may stream from C instead of D. In this case, you now have 2 replicas (C and E) that are missing the data. If you read at quorum, you may now hit C and E instead of B, and get a result that is missing the data. A repair may solve this, as long as your original write went to more than one node, but if your original write only went to one node, you may actually lose data.

Assassinate is a rarely used tool that should only be used tool to force a node out of a cluster. No streaming is performed. The chance of data loss is significantly higher, especially if you use RF < 3 and write with CL < ALL.

Jeff Jirsa
  • 4,391
  • 11
  • 24
  • That makes a lot of sense! Also what would be the way forward to upgrade the servers? Hit a nodetool drain on the node, down it, copy sstables to new node, do a remove node for previous node and add this current node? – Bhuvan Rawal May 09 '16 at 06:53
  • What type of upgrade are you talking about? Hardware upgrade? You probably don't ever need to manually copy sstables. – Jeff Jirsa May 09 '16 at 15:47
  • Ya im wondering what could be the most efficient approach. As per [this link](https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_replace_live_node.html) One needs to add the new node to the server and then after bootstrapping completes, decommission the old node. But how about just drain the old node, stop it, and copy the sstables into upgrade server and get it running? Im sure im missing a point here. – Bhuvan Rawal May 09 '16 at 15:57
  • how to rejoin the node to the cluster after Decommission? Is it rejoinable? – Dai Kaixian Jan 02 '18 at 09:23
  • @DaiKaixian - You have to re-bootstrap it as a new node (so you'd have to wipe all data and commitlogs and add it with a new token). – Jeff Jirsa Jan 02 '18 at 17:52
2

The difference lies in how the data is moved. In decommission, the leaving node moves the data before leaving the cluster where as in remove node, the data is moved using the replicas with the same tokens in the cluster.

Decommission therefore cannot be done using a dead node.

aakhila shaheen
  • 41
  • 5
1

Nodetool removenode , decommission, assassinate, replace. It all depends upon your use case and application reliability. Recommended decommision if you can perform in live cluster than other commands but some times it is also taking more time to stream data and hanging also. then you don't have any option either use removenode command but also some times it is hanging. In this case you have to use assassinate. You can use replace command after streaming all the data from that node. Once the data has been moved to older nodes, the process for removing or replacing is similar for both.

LetsNoSQL
  • 1,478
  • 1
  • 11
  • 23