The Cheapest Reliability Win: How You Deploy

Ask where outages come from and the honest answer surprises people: operator actions — bad config, bad deploys — top the list, well ahead of hardware. That's oddly encouraging. You can't stop disks from dying, but you have total control over how you ship changes. The goal was never "never ship a bad change" — you will. It's to make a bad change cheap: seen by few users, for a short time, and reverted automatically.

Runnable companion: safe-deploys-canary on GitHub. make run ships the same broken release two ways and measures the blast radius (pure Python).

The same broken release, two ways

The simulation ships a release that fails 100% of the requests it serves, against steady traffic of 1000 req/tick (think one tick ≈ one minute):

Exposure to the bad release over time (bar height = % of traffic):

  big-bang             ██████████████████████████████
  canary               ▁▁▁
                       ^deploy

metric                        big-bang      canary
--------------------------------------------------
peak blast radius                 100%          5%
time broken (ticks)                 30           3
total failed requests           30,000         150

• big-bang flips 100% of traffic to the new version at deploy. It stays broken for 30 ticks until a human notices and rolls back by hand — 30,000 failed requests, everyone affected.
• canary sends 5% first. An automated health check sees the error rate spike and rolls back in 3 ticks, before promotion — 150 failed requests, a 5% blast radius.

The release was equally broken in both runs. Canary cut failed requests by 99.5%. The code didn't change — the rollout strategy is what contained the damage.

The two levers

Two independent things made the difference, and you want both:

• Progressive delivery (canary / staged %) shrinks who is affected — blast radius. A bad change reaches a sliver of traffic, not all of it.
• Automated rollback shrinks how long — time-to-recover. Machines detect and revert in the time it takes a human just to notice (3 ticks vs 30 in the sim).

Around those sit the supporting practices: compare the canary against the baseline (not a guessed absolute threshold) so normal variance doesn't cause false trips; keep rollback fast and boring (a button or automatic, never a pressure-authored hotfix); and keep changes backward-compatible so reverting is always safe — a forward-only database migration is what turns a simple rollback into an incident of its own.

Blast-radius thinking

The deeper habit is to treat every risky change as a scoped experiment: who sees it, for how long, and how do we undo it — decided before shipping, not during the incident. That framing is what makes small, frequent deploys safe, which in turn makes each change smaller and safer still. It compounds.

What I'd say in an interview

• How you deploy is a reliability decision — often the cheapest big win, because operator/deploy error is the #1 outage cause.
• Progressive delivery shrinks blast radius; automated rollback shrinks time-to-recover. Use both.
• Think in blast radius for every risky change, and keep changes backward-compatible so rollback is always available.