Retry Storms and Metastable Failure
A brief capacity dip can become a permanent outage — because retries feed the overload that causes them. A runnable simulation of metastable failure, and the cheap defenses that stop it.
The scariest outages aren't the ones caused by a big event — they're the ones where a small event ends and the system stays down anyway. A dependency slows for 30 seconds; an hour later you're still paged, capacity is fine, and nothing is working. That's a metastable failure, and the usual culprit is the most innocent-looking code in your stack: the retry.
Runnable companion:
overload-resilienceon GitHub.make runreproduces the simulation below (pure Python, no services).
The trap
Overload is dangerous because it's self-sustaining. The feedback loop:
- Capacity dips (slow dependency, GC pause, cache flush).
- Requests fail, so clients retry.
- Retries add to incoming load — offered load now exceeds capacity.
- The backend burns its limited capacity re-processing stale retries, so useful throughput (goodput) drops further.
- Lower goodput means more unfinished requests means more retries — back to step 3.
The loop now sustains itself. Even when capacity returns to normal, offered load (new arrivals + a mountain of retries) still exceeds it, so goodput stays near zero. It takes a human — shed load, restart, drain queues — to break out.
Watching it happen
The simulation: a backend serves 100 req/tick when healthy; for 50 ticks its capacity drops to 20; arrivals stay at a comfortable 80/tick. Two client policies react.
Goodput over time (each block ~ 10 req/tick, full = healthy):
naive retries ████████████████████▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
shed+breaker ████████████████████▂▂▂▂▂▂▂▂▂▂██████████████████████████████
^dip starts ^dip ends
phase naive shed+breaker
------------------------------------------------
before dip 80.0 80.0
during dip 0.5 20.0
after dip (recovery) 1.2 80.0
total requests completed naive 8,197 shed+breaker 21,000
The naive policy retries every unserved request, every tick. During the dip, goodput collapses to ~0 — and then stays at ~1/tick long after capacity is restored, because the retry backlog keeps offered load far above capacity. It completes 8,197 requests and needs a human to recover.
The shed+breaker policy caps offered load at roughly what the backend can serve and rejects the rest fast (cheap) instead of retrying. Capacity is never wasted, the backlog can't snowball, and the instant the dip ends it snaps back to 80/tick on its own — completing 21,000 requests (~2.6x) and self-healing.
Same dip. The only difference is how the client handles failure.
The cheap defenses
| Technique | What it does |
|---|---|
| Load shedding | Reject excess fast and cheap at the edge, so overload can't consume capacity |
| Circuit breaker | When a dependency is failing, stop calling it for a cooldown — fail fast, give it room to recover |
| Backoff + jitter | Space retries out and de-synchronize them, so they don't return as a thundering herd |
| Backpressure | Bounded queues that propagate "slow down" upstream instead of absorbing unbounded work |
| Retry budgets | Cap retries to a small fraction of traffic so they can't amplify load |
The common thread: bound the offered load, and fail fast. A retry without these isn't resilience — it's an amplifier.
What I'd say in an interview
- Overload is self-sustaining: a brief trigger can cause a lasting outage through a retry feedback loop. The term is metastable failure.
- The cure is to bound offered load (shed early, cap retries) and fail fast (circuit breaker), so capacity is never wasted on a retry storm and the loop can't form.
- Backoff with jitter, backpressure, and retry budgets complete the toolkit. Unbounded retries are how you turn a blip into an outage.