The Partition Key Decision You Can't Take Back

Sharding is the heavy-machinery answer to scale — reach for it only when a single node genuinely can't hold the data or the write throughput (a read bottleneck wants a read replica, not a shard). Once you do shard, one decision dominates all others: the partition key. Know it, and a request goes straight to one shard; don't, and it scatters across all of them — and the scheme is very hard to change once data exists. Here are the three trade-offs that decision turns on, simulated so "pick a good key" stops being hand-waving.

Runnable companion: partition-key-strategies on GitHub. make distribution && make consistent && make hotkey reproduces every number below (pure Python).

1. Hash vs key-range: hot spots vs range scans

The two classic schemes trade the same property in opposite directions. Where does the current write load land (8 shards, fair share = 12.5%)?

workload                scheme        busiest shard     skew
------------------------------------------------------------
uniform keys            key-range             12.6%     1.01
uniform keys            hash                  12.6%     1.01
monotonic (timestamp)   key-range            100.0%     8.00
monotonic (timestamp)   hash                  12.6%     1.01

A monotonically increasing key (timestamp, auto-increment id) is the classic trap: under key-range every new write lands in the top range, so one shard takes 100% of current writes while the other seven idle. Hash scatters them and stays even — but destroys ordering, so a range scan pays for it:

Range scan over a contiguous 1% slice:
  key-range      1 shard touched
  hash           8 shards touched   (scatter/gather across all of them)

Key-range answers a range scan from one shard; hash turns it into a scatter/gather whose latency is the slowest shard and doesn't improve as you add shards. The escape hatch is a compound key — hash a prefix (user_id), sort by the rest (timestamp): writes stay balanced (skew ~1.06) and a per-user time-range scan hits one shard. You just can't scan across users by time cheaply. Pick the partition key for the query that must be fast.

2. Virtual nodes even out keys, not load

Consistent hashing places nodes on a ring; with one point per node the ring is lumpy. Virtual nodes average the arcs:

 vnodes/node   key skew
 -------------------------
          1       3.20
         16       1.56
        256       1.12

Skew falls from 3.20 → 1.12. But an even key split is not an even load split, because real access is skewed (Zipf):

key-distribution skew (keys each node owns)     1.12
request-LOAD skew (skewed access to them)       1.84

Keys are balanced (1.12) yet request load is lopsided (1.84). The ring can't fix that — the imbalance lives inside the keys.

3. The hot key: split it or melt a shard

One celebrity key (a viral post, a top-seller's stock counter) taking 30% of traffic is indivisible — it all hashes to one shard. Split it into K sub-keys with a random suffix and its traffic fans out (16 shards, fair share = 6.2%):

 hot-key splits  busiest shard  read fan-out
 -------------------------------------------
             1          34.3%             1
             4          12.0%             4
            32           8.2%            32

Undivided, the hot key pins one shard at ~34% — ~5x its fair share — while 15 idle. Splitting drops it back toward fair. The catch: it fans out writes cleanly but forces every read to gather all K sub-keys (read fan-out = K), plus bookkeeping of which keys are split. So you split only known-hot keys, not everything — with diminishing returns and sub-key collisions past K=4.

What I'd say in an interview

• Shard only when one node can't hold the data or the write rate — read-bound → replica. Don't shard early; sharding adds cross-shard transactions, routing, and rebalancing.
• The partition key is near-irreversible — choose it for the read path that must be fast; changing it moves most of the data.
• Hash vs key-range = range scans vs hot spots. A monotonic key + key-range → one hot shard; hash fixes writes but scatters scans; a compound key buys back locality.
• Virtual nodes even keys, not load. A hot key survives every balancing trick — isolate it, cache it, or split it with a random suffix and accept the read fan-out.