docs/solutions/best-practices/gate-rollouts-on-traffic-weighted-data-not-a-hand-picked-cohort.md
Rama main · réplica de lectura del repositorio original.
---
title: "Gate a rollout on traffic-weighted data, not a hand-picked cohort"
date: 2026-07-18
category: best-practices
module: "Bootstrap storage cutover verification (U-K3 gate, #5338)"
problem_type: best_practice
component: development_workflow
severity: medium
applies_when: "Writing the pass/fail gate for a migration, cutover, or A/B rollout — especially when the plan names a specific 'worst case' cohort to gate on"
tags: [verification-gate, rollout, percentile, sample-size, traffic-weighted, decision-criteria, measurement, cutover, p95]
---
# Gate a rollout on traffic-weighted data, not a hand-picked cohort
## Context
The plan for cutting the bootstrap public tier over to a new storage backend specified its verify
gate up front as: *"gate on the worst APAC cohort (hkg1/syd1/bom1/sin1) — the candidate's p99 must
be materially below the incumbent there."* Picking the worst-looking region and demanding the
candidate win there feels like the conservative, rigorous choice.
It was neither. When the measurement window actually ran, that gate produced a table nobody could
act on:
- **APAC is a minority of the traffic.** The user base was genuinely worldwide — the top countries
by volume were the US, India, Italy, Australia, Germany, the UK, Japan, France, Singapore, and
the UAE. Gating on four APAC cities decided a global cutover on a slice of it.
- **One of the four named cities never gathered enough data to judge.** Seoul finished the window
with roughly 14–20 samples per side. It reported a "win" that meant nothing.
- **A per-city p95 on a thin cell is noise, not a verdict.** With ~20 samples, the 95th percentile
*is* essentially the maximum observed value, so one slow read moves it arbitrarily.
Jakarta made the last point unmissable. Its p95 read **222 ms at n=4, then 1418 ms at n=32, then
1341 ms at n=56, then 799 ms at n=86** — a ~6× swing driven by sample count alone, before it
settled. A gate evaluated at any single one of those moments would have returned a confident and
different answer.
## Guidance
**Pick the gate's denominator from where users actually are.** Replace "does the candidate win in
cohort X" with a single traffic-weighted metric over *all* traffic — here, *the share of all reads
clearing the client latency budget*. Weighting is automatic, no cohort has to be chosen, and the
number is directly meaningful to the decision being made.
Then structure the gate as **two separate tests**, because they answer different questions:
1. **Relative (the blocker):** does the candidate regress anyone? Compare candidate vs incumbent
per region, *but only in cells with enough samples to judge*. A regression in a well-sampled
region blocks the rollout.
2. **Absolute (the quality bar):** what share of all traffic clears the target? Report it; hold it
to a threshold. This is a bulk-quality measure, **not** a per-region hard requirement.
**Add a minimum-sample threshold so thin cells report `inconclusive` rather than a false PASS or
FAIL.** This is the single highest-value line in a gate script. Without it, small cells emit
confident verdicts that flip run to run.
Finally, keep the absolute bar from masquerading as a blocker. A region where the candidate is
*better than the incumbent yet still misses the absolute target* is a follow-up, not a stop —
because shipping still improves those users. Conflating the two is what made the first version of
this gate hard-fail on a 32-sample remote city while the candidate was beating the incumbent there.
## Why This Matters
Reframing from the APAC cohort to a global traffic-weighted metric did two things.
It produced a **decisive** answer where the cohort gate had produced an ambiguous one: the
candidate cleared the budget for **99.6 %** of all reads versus the incumbent's **84.4 %** — the
incumbent was missing roughly one request in six, globally.
More importantly, it **surfaced the findings the cohort lens had hidden entirely**:
- The incumbent was *catastrophic* in regions nobody had named as the worst case — clearing the
budget for only ~40 % of reads in Africa and ~39 % in Oceania. The APAC-only gate would never
have looked there.
- The incumbent's platform had **no Middle East region at all**, so MENA users were being served
from Europe — while the candidate served them locally at ~270 ms with 100 % clearing the budget.
For a MENA-centred user base this was arguably the single most valuable finding of the exercise,
and the original gate could not have produced it.
The hand-picked cohort did not just measure the wrong thing — it pointed attention away from where
the real problem was.
## When to Apply
Apply whenever writing the accept/reject criteria for a cutover, migration, dependency swap,
infrastructure change, or performance A/B — particularly when a plan or ticket already names a
"worst case" cohort to gate on. Treat that named cohort as a *diagnostic view*, not the gate.
Also apply when reviewing a gate someone else wrote. The tells:
- The gate's denominator is a chosen subset rather than all traffic.
- Percentiles are computed per narrow cell with no sample-count guard.
- A single threshold serves as both "did anyone regress" and "is quality good enough".
Less relevant when the cohort genuinely *is* the population (a region-specific launch), or when
per-cell volumes are large enough that thin-cell noise cannot arise.
## Examples
**Before — cohort gate with a hard absolute clause and no sample guard:**
```python
APAC = ["hkg1", "syd1", "bom1", "sin1"]
for city in APAC:
ok = kv_p95[city] < redis_p95[city] and kv_p95[city] < BUDGET_MS
if not ok:
gate_pass = False # a 32-sample city can hard-fail the whole rollout
```
**After — traffic-weighted headline, sample-guarded relative test, separate quality bar:**
```python
MIN_SAMPLES = 25 # below this a per-cell p95 sits on the near-max sample => inconclusive
BULK_TARGET = 99.0 # % of ALL reads that must clear the budget
# 1. Relative test (the blocker) — only where the cell can actually be judged.
for city in regions:
if kv_n[city] < MIN_SAMPLES or rds_n[city] < MIN_SAMPLES:
verdict[city] = "thin" # inconclusive, NOT a pass or fail
elif kv_p95[city] > rds_p95[city]:
regressions.append(city) # this is what blocks
elif kv_p95[city] > BUDGET_MS:
budget_misses.append(city) # beats incumbent but misses target => follow-up
# 2. Absolute quality bar (traffic-weighted over ALL reads, no cohort chosen).
under_pct = 100.0 * reads_under_budget / reads_total
gate = (not regressions) and under_pct >= BULK_TARGET
```
The rewritten gate changed the reported outcome without any change in the underlying data: the
same window that "FAIL"ed on a 32-sample remote city under the first version returned a clean pass
with that city correctly reported as *"beats incumbent, misses absolute target — follow-up"*, and
the thin cell reported as *inconclusive* instead of a fabricated win.
## Related
- `docs/solutions/2026-07-16-bootstrap-kv-verify.md` — the verify artifact this gate produced,
including the continent-level table that the cohort framing would have omitted.
- `docs/solutions/design-patterns/hedge-cache-read-against-origin-not-hard-timeout.md` — the same
window's data used for a different decision, and another instance of the general habit: measure
the distribution before committing to a constant.