Measure Latency Right: Percentiles, Not Averages

"Average response time is 8 milliseconds" is one of the most common sentences in a design review, and one of the most misleading. Performance is a distribution, not a number — and the requests that hurt are the slow ones, which land disproportionately on your most active, most valuable users. Here are three tiny, runnable demos (pure standard library, no services) that make the point concrete.

Runnable companion: measure-latency-right on GitHub. make percentiles && make aggregation && make coordinated reproduces every number below.

1. The mean lies

Take a typical service latency distribution — a fast body with a thin slow tail — and print the mean next to the percentile ladder:

  mean               8.2   <- what a dashboard 'average' shows
  p50 (median)       3.4   <- the typical request
  p95               20.2
  p99               59.8   <- 1 in 100 requests
  p99.9            466.3   <- 1 in 1,000 requests
  max              499.9

The mean (8.2 ms) hugs the median and hides a p99 ~7x worse and a p99.9 ~57x worse. This matters more than it looks, because of fan-out: a page that makes 100 backend calls waits on its slowest one, so its effective latency tracks a high percentile of the per-call distribution. Rare slowness becomes common at the page level. Set SLOs on p95/p99/p999 — never on the mean.

2. You can't average percentiles

Another routine mistake: two nodes each report a p99, and a dashboard shows the average of the two. Combine a healthy node and an overloaded one, each serving 50k requests:

  node A p99                     59.6 ms
  node B p99                    775.7 ms
  WRONG: average of the two     417.6 ms
  RIGHT: merged histogram       751.0 ms
  ground truth (all samples)    751.8 ms

Averaging the two p99s is off by 44%. A percentile is not a linear operator — you cannot rebuild the combined distribution from two summary numbers. The right way is to keep histograms (counts per latency bucket), add the bucket counts, and read the percentile off the merged histogram; that recovers the true value exactly. This is precisely why metric systems ship mergeable histograms (HdrHistogram, Prometheus, DDSketch) instead of pre-computed percentiles. Store distributions, not summaries — summaries can't be combined.

3. Coordinated omission — your load test is lying too

The subtlest one. Simulate a server that freezes for 500 ms and measure it two ways:

  closed-loop  samples= 19000  p50=  0.5  p99=    0.5  p99.9=    0.5  max=  500.5  (ms)
  open-loop    samples= 10000  p50=  0.5  p99=  450.5  p99.9=  495.5  max=  500.5  (ms)

Same stall, opposite stories. A closed-loop load generator — send a request, wait for the response, send the next — blocks on its single in-flight request during the freeze and simply stops issuing new ones. It records one slow sample and reports a perfect p99 of 0.5 ms. Meanwhile an open-loop measurement keeps sending at the intended rate, ~1,000 requests pile into a backlog, and it reports the truth users feel: p99 of 450 ms.

This is coordinated omission: the client coordinates with the server to omit exactly the samples that would have been slow. It's why a benchmark can show wonderful latencies that evaporate in production. The fix: measure at the client, drive load open-loop, or apply a coordinated-omission correction (wrk2 and HdrHistogram do this).

What I'd say in an interview

• Latency is a distribution. The mean is nearly useless for it — report percentiles, and watch the tail, because fan-out amplifies it.
• Percentiles don't average. Merge histograms to combine across nodes or time; store distributions at the edge, not summaries.
• Beware coordinated omission — a load tool that waits for responses hides the tail. Measure open-loop, at the client.