In raft, when a node restart, it try to redo all the log entries to catch up the state. But if node goes down again in recovery phase, node would do some op twice. These twice redo op will violate state machine if ops are not idempotent.
According to the description above, my question is, is that necessary to make ops idempotent in a system using raft in practice?
In my experience Raft, Paxos, and friends are used to implement distributed state machines and databases (which is a distributed state machine). When viewed in this context you really, really, really want the entries to be idempotent; otherwise your state machines will diverge sooner than later.
You want idempotency even if you were using non-ordered queues---like rabbit-mq or ActiveMQ---because their guarantees are at best at least once delivery.
Sure, there's nothing preventing you from having non-idempotent entries. Except, perhaps a design review. In short, I cannot think of one scenario where a queue is better served with non-idempotent entries. Not one.
No, there is no requirement for log entries to be idempotent if you are using a distributed consensus algorithm such as Paxos or Raft. There are many advantages to designing your system around idempotent operations, but it is not always possible to do so. Distributed consensus algorithms are a great fit for cases where you cannot achieve idempotence so that you need to be sure that operations are processed at most once on each replica. Moreover they let you ensure that the operations are always processed in exactly the same order on each replica. These are very strong properties, and you need to perform some relatively-expensive coordination to ensure that they hold, which is why we try and avoid them wherever possible.
Distributed consensus guarantees that each log entry will be the same on every replica as long as that log entry has been accepted by a majority of the replicas. Each replica must only process log entries that have been accepted by a majority of replicas, because other log entries might change during a recovery. Each replica must carefully keep track of which operations it has already processed to avoid processing them again during a recovery. This is simple to do since the log is totally ordered, so each replica can track the operations it has processed with a single number representing the position of the last processed operation in the log.
In fact, another way of looking at distributed consensus is that it is an effective way to add idempotence (and commutativity) back into a collection of operations that might not be idempotent (or commutative). So no, the operations don't need to be idempotent because you can obtain the idempotence you need from the distributed consensus algorithm instead.
Yes they need to be idempotent and deterministic in the sense that executing the operation on the slave should move the slave to the same state as the master.
This mean operation cannot read external state (wall clock time, random number generator) but can only use the current state machine value and the data from the log entry to decide on the next state machine value.
But because of the design of the raft protocol it provide at-most-once delivery semantic thus, its not possible for one operation to be executed or written to the replication log several time.
You do not need to check that this operation already got executed like you would in a system that only provide at least once delivery semantic
