Split-brain simulator
When a network partition cuts a cluster in two, each side may decide the other is dead and carry on alone. Split the nodes your way, choose how leaders are elected, and see which side keeps working, which writes are refused, and which acknowledged writes disappear when the network heals.
- Side A (1, 2) has the old leader, node 1, but only 2 of 5 votes: it can no longer reach a quorum of 3, so its writes fail.
- Side B (3, 4, 5) hears nothing from the leader for 1000 ms and elects node 3 with 3 of 5 votes.
- On healing, the minority nodes catch up from the majority’s log. No acknowledged write is lost, because none was acknowledged without a majority.
| Side | Votes | Acked | Refused | Lost |
|---|---|---|---|---|
| A | 2 | 0 | 100 | 0 |
| B · leader | 3 | 90 | 10 | 0 |
| Policy | Leaders | Acked | Lost |
|---|---|---|---|
| Majority quorum | 1 | 90 | 0 |
| No quorum check | 2 | 190 | 100 |
| No quorum, last write wins | 2 | 190 | 20 |
How to use the simulator
Choose the number of nodes and click nodes to move them between sides A, B and C of the partition; dashed lines are the links the partition cut. Pick which node led before the split and how leaders are chosen. During the partition, clients on every side keep sending writes; the tables show how many each side acknowledged or refused, and how many acknowledged writes were lost when the sides reconnected.
What is split brain?
Many systems need exactly one node in charge: the primary database, the holder of a lock, the leader that orders writes. They detect a dead leader by silence, so a node that stops hearing from the leader cannot tell whether the leader crashed or the network between them failed. If a partition splits the cluster and each side promotes a leader of its own, the system has split brain: two leaders accepting writes that the other never sees.
When the network heals, the two histories conflict. Some writes that clients were told had succeeded have to be thrown away or merged, and the clients never find out.
Majority quorums
The standard defence is a majority quorum: a leader is elected only with votes from more than half of the cluster, and a write is committed only once more than half have stored it. Two disjoint groups cannot both contain more than half of the nodes, so at most one side of any partition can make progress. Raft, Paxos-based systems, ZooKeeper and etcd all work this way.
In the default setup, 5 nodes split 2 and 3 with the old leader on the small side. Under a majority quorum, the old leader can no longer commit and refuses writes, while the 3-node side elects a new leader after its election timeout. Over a 10-second partition with 10 writes a second sent to each side, 90 writes were acknowledged, 110 refused and 0 lost. Without the quorum check, both sides kept accepting writes: 190 acknowledged, and 100 of them thrown away on healing when every node followed the newer leader.
The price is availability: clients on the minority side get errors for the whole partition. That is the trade the CAP theorem describes, choosing consistency over availability during a partition.
Cluster sizes and witnesses
| Nodes | Majority quorum | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 6 | 4 | 2 |
| 7 | 4 | 3 |
A cluster of n nodes needs ⌊n/2⌋ + 1 for a quorum and survives ⌊(n − 1)/2⌋ failures. Going from an odd number to the next even one adds a machine without adding tolerance, and makes an exact half-and-half split possible: 4 nodes split 2 and 2 leave neither side with 3 votes, so the cluster had 0 leaders and refused all 200 writes. A witness (also called an arbiter or tie-breaker) is a member that votes but stores no data. Adding one to side B of the same split gave that side 3 of 5 votes and 1 leader again.
Last write wins
Some systems choose availability instead: every side keeps accepting writes, and when the partition heals each key keeps the write with the latest timestamp. Nothing is refused, but on every key that both sides wrote, one side’s acknowledged value silently disappears. Timestamps also come from different machines’ clocks: in the default setup with side B’s clock running 200 ms fast, 20 keys lost a value one side had acknowledged, and on 1 of them the value kept was in fact the older write.
Fencing
Quorums stop two leaders being elected, but an old leader can still act on stale belief: a process that paused (for garbage collection, say) may wake up still thinking it holds a lease that has expired. Fencing stops it from doing damage. With fencing tokens, every new leader or lock holder gets a number larger than the last, and the storage rejects any write carrying a token lower than one it has already seen, a scheme Martin Kleppmann describes in his analysis of distributed locking. Older systems fence at the hardware level, cutting the old primary’s power or storage access before promoting a new one (STONITH: shoot the other node in the head).
Split-brain questions and answers
What is split brain?
A failure in which a cluster that should have one leader or primary ends up with two or more, usually because a network partition left each group of nodes believing the others are dead. Both sides accept writes, and when the network heals their histories conflict, so some acknowledged writes are lost or must be merged by hand.
How does a quorum prevent split brain?
A side may only act with more than half of all votes, and two disjoint groups cannot both have more than half. With 5 nodes split 2 and 3, only the 3 can elect a leader; the old leader on the 2 side can no longer commit. In this page’s run, the majority quorum lost 0 acknowledged writes, against 100 without a quorum check.
Why do clusters have an odd number of nodes?
Because adding a node to an odd-sized cluster doesn’t let it survive more failures: 3 nodes tolerate 1 failure and so do 4, 5 tolerate 2 and so do 6. An even cluster can also split exactly in half, leaving no side with a majority. A witness, a member that only votes, breaks such ties cheaply.
What is a fencing token?
A number that increases every time a new leader or lock holder is chosen, passed along with every write. The storage remembers the highest token it has seen and rejects writes carrying a lower one, so an old leader that hasn’t noticed it was replaced can’t overwrite newer data.
Is last write wins safe?
It always resolves conflicts, but by throwing away all but one write per key, and "last" is decided by timestamps from different machines. In this page’s run with side B’s clock 200 ms fast, 20 keys lost a value one side had acknowledged, and 1 kept a write that actually happened earlier.