Quantifying "Eventual": Replication Consistency, Measured

Replication is where "it works on one node" collides with concurrency, node death, and network delay — and all the difficulty is in propagating changes correctly. The decisions (single vs multi vs leaderless leader, sync vs async, quorum sizes, failover policy) have precise, measurable consequences on correctness. Here are three of them, simulated so the trade-offs stop being abstract. (Reminder: replication is not backup — it faithfully replicates your mistakes too.)

Runnable companion: replication-consistency on GitHub. make quorum && make anomalies && make failover reproduces every number below (pure Python).

1. w + r > n is a probability, not a guarantee

Leaderless replication (Dynamo-style): write to w of n replicas, read from r. The rule w + r > n forces the read and write sets to overlap. Does it guarantee a fresh read?

 w  r  w+r  w+r>n?   stale% (strict)   stale% (10% drop)
 1  1    2      no             80.0%               82.3%
 2  2    4      no             29.8%               36.2%
 2  3    5      no              9.8%               16.1%
 3  3    6     yes              0.0%                3.5%
 4  4    8     yes              0.0%                0.1%
 1  5    6     yes              0.0%               10.0%

With a strict quorum, the instant w + r > n stale reads hit 0%. But add a 10% chance a write silently misses a node (a dropped message, a node briefly down) and even w+r>n leaks stale reads. Real systems have more such holes (sloppy quorums, concurrent writes, read repair) — all adding staleness, not removing it. So w + r > n is a dial on the probability of freshness, not a guarantee. If the number matters, measure your actual staleness; if you need certainty, use a linearizable store.

2. Eventual consistency has a specific shape

Lagging replicas don't just make data "a bit old" — they break guarantees users feel:

anomaly                 naive routing  with fix   fix
read-your-writes                75.0%      0.0%   read your own writes from the leader
monotonic reads                 18.7%      0.0%   pin the user to one replica (sticky)
consistent prefix               18.6%      0.0%   keep causal writes on one partition

• Read-your-writes — you post a comment, refresh, and it's gone (you hit a replica without your write).
• Monotonic reads — a refresh shows older data than a moment ago (two replicas, different lag; time goes backward).
• Consistent prefix — you see an answer before the question it replies to (causal writes on differently-lagging partitions).

Each has a known fix that drives it to 0. For a complex app, a strongly-consistent database deletes the whole class of bugs — the NewSQL argument.

3. Failover silently loses acked writes

strategy                                  writes lost   ack latency
async, promote freshest follower                    2    ~1x (fast)
async, promote an arbitrary follower               12    ~1x (fast)
semisync (ack after >=1 follower has it)            0           ~2x
sync (ack after ALL followers have it)              0    ~Nx (slow)

The leader ack'd 1,000 writes, then crashed. With async replication, any ack'd write the new leader never received is silently lost — clients were told "saved". Two rules follow: always promote the freshest follower (loss 2 vs 12), and use semisync/quorum-sync when writes must not be lost (0 loss, at a latency cost). The cardinal sin is treating async as if it were sync. (Not shown but real: split-brain if two leaders are promoted, and picking the failover timeout — which is why many teams keep failover manual.)

What I'd say in an interview

• Pick the topology by constraints: unique/invariant enforcement → single-leader; multi-region/offline → multi-leader; max resilience, tolerate stale → leaderless. (Multi-leader and leaderless can't enforce global invariants like a non-negative balance.)
• w + r > n is a probability, not a guarantee — measure real staleness.
• Eventual consistency has named anomalies — read-your-writes, monotonic, consistent prefix — each with a fix, or buy strong consistency to remove them.
• Async can lose acked writes on failover — promote the freshest follower, use semisync/quorum-sync for durability, and never pretend async is sync.