There are two ways one might structure a system:
- all nodes may have the same responsibilities.
- nodes may have separate, distinct roles.
Consensus algorithms for replication generally opt for having distinct roles for each node.
Having a single fixed leader or master server is an optimization that makes the system more efficient, since we know that all updates must pass through that server. Nodes that are not the leader just need to forward their requests to the leader.
Note that having distinct roles does not preclude the system from recovering from the failure of the leader (or any other role). Just because roles are fixed during normal operation doesn’t mean that one cannot recover from failure by reassigning the roles after a failure (e.g. via a leader election phase). Nodes can reuse the result of a leader election until node failures and/or network partitions occur.
Both Paxos and Raft make use of distinct node roles. In particular, they have a leader node (“proposer” in Paxos) that is responsible for coordination during normal operation. During normal operation, the rest of the nodes are followers (“acceptors” or “voters” in Paxos).
During normal operation, a partition-tolerant consensus algorithm is rather simple. As we’ve seen earlier, if we didn’t care about fault tolerance, we could just use 2PC. Most of the complexity really arises from ensuring that once a consensus decision has been made, it will not be lost and the protocol can handle leader changes as a result of a network or node failure.
All nodes start as followers; one node is elected to be a leader at the start. During normal operation, the leader maintains a heartbeat which allows the followers to detect if the leader fails or becomes partitioned.
When a node detects that a leader has become non-responsive (or, in the initial case, that no leader exists), it switches to an intermediate state (called “candidate” in Raft) where it increments the term/epoch value by one, initiates a leader election and competes to become the new leader.