Fault vs Failure: Killing a Replica on Purpose

Reliability has a precise definition that's easy to say and hard to live: keep faults from becoming failures. A fault is one component misbehaving — a node dies, a disk fails, a link drops. A failure is the whole system not delivering its service. You can't prevent faults (hardware dies, dependencies wobble, at scale it's constant), so the entire discipline is making sure any single fault is absorbed. This lab proves it the only honest way: by killing something and watching.

Runnable companion: chaos-failover on GitHub. make run starts two replicas, kills one mid-traffic, and measures both client strategies.

The experiment

Two identical stateless replicas run behind the test. The same run measures two client strategies against the same kill:

• SPOF — always calls replica-1 only (a single point of failure).
• HA — tries replicas in order, marks a failed one down, and fails over to the next (what a load balancer or a smart client does).

Halfway through, we docker kill replica-1. The result:

Availability (% of probes served):

  phase                  SPOF (1 node)   HA (failover)
  ----------------------------------------------------
  before kill                      98%            100%
  after kill                        0%            100%

  downtime after kill   SPOF   3.5s      HA  0.0s   (of a 3.5s window)

Timeline (● served, · failed) — kill happens at the marker:

                                                       ▼ kill
  SPOF  ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●·······················
  HA    ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

One fault, two outcomes. For the SPOF client the fault becomes a failure — 0% availability, downtime that only ends when a human brings the replica back. For the HA client the very next request fails over to replica-2; availability stays at 100%, downtime 0.0s. The only difference is redundancy plus a failover policy.

What actually makes it survive

Fault tolerance needs two ingredients, and both are required:

1. Redundancy — more than one instance, with no shared fate (not two containers on the same host, disk, or power feed).
2. A failover policy — health checks / fast failure detection plus rerouting. Spare capacity nobody routes to is not fault tolerance; it's just a bigger bill.

A single point of failure is any component whose loss takes the system down: a lone database, one load balancer, a single availability zone, a shared config service. Hunting and removing SPOFs is the first, highest-leverage reliability move — because your availability is capped by your weakest single-instance component, no matter how clever everything else is.

Why kill things on purpose

The experiment doesn't argue that failover works — it injects the fault and measures the outcome. That's chaos engineering: faults are inevitable, so trigger them deliberately, in a controlled way, and verify the system tolerates them before production does it for you at 3am. An untested failover path is a hypothesis, not a guarantee — config drift, a broken health check, or DNS caching that defeats rerouting all hide until something dies. Start simple (kill a process), then escalate to latency, packet loss, partial failures and zone loss.

What I'd say in an interview

• Define reliability precisely: stop faults from becoming failures. Faults are routine at scale; a failure is a design outcome.
• Hunt SPOFs first — one unreplicated component caps your availability.
• Redundancy needs a failover policy, not just spare instances — and stateful failover (a DB primary) is much harder than this stateless demo, trading consistency for availability.
• Inject faults on purpose. An untested failover is a guess; a tested one is a guarantee.