Source: Production A/B testing methodology at major web search and e-commerce companies; statistical literature on online experiments
Classification — End-to-end pattern for proposing, running, analyzing, and deciding on search changes via controlled online experiments.
Convert proposed search changes into shipped improvements (or learned-from failures) via the discipline of controlled experimentation, with statistical rigor that distinguishes real effects from noise.
Search changes look promising in development but their production effect is often different. Synonym additions that seem clearly helpful may not move metrics; ranking adjustments that look minor may have substantial impact; query understanding changes may help one segment while hurting another. The discipline of A/B testing makes these dynamics visible; without it, the team ships changes based on intuition and never learns what actually works.
Hypothesis registration. Before running the test, write down: the change being tested; the primary metric expected to move; the expected magnitude ("+1–2% CTR for queries containing X"); the segment expected to be most affected; the guardrail metrics that must not regress. The registration is auditable; the test's success criterion is the pre-registered metric meeting the pre-registered threshold. Production patterns: maintain a registry (spreadsheet, internal tool) of all tests with their registrations; revisit the registry monthly to evaluate the team's hit rate (what fraction of tests show their expected effect?).
Power calculation. Compute the sample size needed to detect the expected effect at the desired statistical power. For a binomial metric (CTR) with baseline rate p and expected lift d, the required sample size per arm scales roughly as: n ≈ 16 × p × (1−p) / d² (at 80% power, 5% significance). For CTR baseline 20% and lift 1%: n ≈ 16 × 0.2 × 0.8 / 0.0001 = 25,600 queries per arm. For continuous metrics (NDCG, dwell time), use t-test sample-size formulas. Most A/B testing platforms (Google Optimize, Optimizely, internal builds) provide power calculators; the discipline is using them.
Bucket assignment. Users (or sessions, or queries) get assigned to test buckets via deterministic hashing on a stable identifier. The hash ensures that the same user sees consistent behavior throughout the test (and across multiple tests, if appropriate). Production patterns: hash on user_id where users are logged in; hash on session_id otherwise; document the assignment unit explicitly (user-level vs session-level vs query-level matters for analysis).
Ramping. Tests typically ramp gradually before reaching full sample: 1%, 5%, 25%, 50%. The ramp serves two purposes: catches catastrophic regressions before they affect many users (guardrail metrics monitored during ramp); reveals scale-dependent effects (some changes work at small scale but not at large). At each ramp level, the team holds for a defined period (24–48 hours typical) and reviews guardrails before advancing. The ramp can be paused or reverted if guardrails fail.
Guardrail monitoring. During the test, monitor guardrail metrics for the test arm. Common guardrails: zero-result rate (must not increase substantially); latency p95 (must not increase); revenue per session (for e-commerce; must not decrease); specific subgroup metrics (the change should help the target segment without harming others). Guardrail violations during ramp pause the test for investigation; not every guardrail violation kills a test (some are noise or expected) but each warrants review.
Analysis. After reaching the pre-calculated sample size: compute the primary metric for control and treatment arms; compute the statistical significance (typically using t-test or bootstrap, depending on metric properties); check the segment analyses identified in the registration. The analysis follows the registration: significance on the pre-registered metric with the pre-registered direction is the success criterion. Production patterns: don't peek at significance during the test (peeking inflates false positive rate); use sequential testing methods if you genuinely need early stopping; document the analysis methodology so it's consistent across tests.
Decision-making. Three outcomes: ship (primary metric moved significantly in the desired direction, guardrails held); kill (primary metric didn't move or moved wrong direction); iterate (mixed results suggesting the approach has merit but needs adjustment). The decision follows the pre-registered criteria. Production discipline: write a decision document with the test outcome, the decision rationale, and what was learned. The document is auditable and feeds future test design.
Common failures. Underpowered tests — too small to detect realistic effects. Peeking — stopping when p<0.05 first arises rather than at the pre-registered sample size; inflates false positive rate substantially. Metric-only optics — declaring success based on metrics while missing user-experience issues that metrics don't capture. Multiple comparison without correction — testing many metrics and declaring success on any that moved; needs Bonferroni or similar correction. Segment cherry-picking — looking for any subgroup where the change helped, after the primary metric failed. The disciplines for avoiding each are well-established; production teams encode them in their testing infrastructure.
Most production search changes — ranking adjustments, query understanding updates, schema changes, synonym additions, analyzer changes. The discipline applies broadly; the only changes that don't need A/B testing are bug fixes (where the right answer is unambiguous) and infrastructure changes that shouldn't affect search quality (where guardrails are sufficient).
Alternatives — offline-only evaluation for the rare changes where the judgment set is sufficient (and the team accepts that production behavior may differ). Pre/post analysis for changes that can't be A/B tested (rare in modern infrastructure); pre/post is less rigorous than A/B because it doesn't control for confounding changes.
- Kohavi, Tang, Xu, "Trustworthy Online Controlled Experiments" (Cambridge, 2020) — the canonical text
- Production methodology writings from Google, Microsoft, Etsy, Bing on search A/B testing
- Statistical literature on power calculation and sequential testing
Code
# Search A/B test analysis with power calculation, significance
testing, and segments
import numpy as np
from scipy import stats
import pandas as pd
def required_sample_size(baseline_rate: float, expected_lift: float,
alpha: float = 0.05, power: float = 0.80) -> int:
"""Compute required sample size per arm for binomial metric (e.g.,
CTR).
Returns sample size per arm needed to detect the expected lift
at the given significance level and power.
"""
p1 = baseline_rate
p2 = baseline_rate + expected_lift
pooled = (p1 + p2) / 2
z_alpha = stats.norm.ppf(1 - alpha / 2)
z_beta = stats.norm.ppf(power)
numerator = (z_alpha * np.sqrt(2 * pooled * (1 - pooled)) +
z_beta * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
denominator = (p2 - p1) ** 2
return int(np.ceil(numerator / denominator))
# Example: detect 1% absolute lift on 20% baseline CTR
n = required_sample_size(baseline_rate=0.20, expected_lift=0.01)
print(f"Sample size per arm: {n:,}")
# Output: Sample size per arm: ~25,000
def analyze_ab_test(
df: pd.DataFrame, # columns: variant (\'control\' / \'treatment\'),
clicked (bool)
metric_col: str = \'clicked\',
segment_col: str | None = None,
) -> dict:
"""Compute test results: lift, significance, confidence interval.
Optionally segment by another column.
"""
results = {}
def compute_stats(group_df):
control = group_df[group_df.variant ==
\'control\'][metric_col].values
treatment = group_df[group_df.variant ==
\'treatment\'][metric_col].values
if len(control) == 0 or len(treatment) == 0:
return None
c_rate = control.mean()
t_rate = treatment.mean()
lift_abs = t_rate - c_rate
lift_rel = lift_abs / c_rate if c_rate > 0 else None
# Two-proportion z-test
pooled = (control.sum() + treatment.sum()) / (len(control) +
len(treatment))
se = np.sqrt(pooled * (1 - pooled) * (1/len(control) +
1/len(treatment)))
z = lift_abs / se if se > 0 else 0
p_value = 2 * (1 - stats.norm.cdf(abs(z)))
# 95% CI on lift
se_lift = np.sqrt(c_rate*(1-c_rate)/len(control) +
t_rate*(1-t_rate)/len(treatment))
ci_low = lift_abs - 1.96 * se_lift
ci_high = lift_abs + 1.96 * se_lift
return {
\'control_n\': len(control),
\'treatment_n\': len(treatment),
\'control_rate\': c_rate,
\'treatment_rate\': t_rate,
\'lift_abs\': lift_abs,
\'lift_rel\': lift_rel,
\'p_value\': p_value,
\'ci_95\': (ci_low, ci_high),
\'significant\': p_value < 0.05,
}
# Overall analysis
results[\'overall\'] = compute_stats(df)
# Per-segment if requested
if segment_col:
for segment, sub_df in df.groupby(segment_col):
results[f\'segment_{segment}\'] = compute_stats(sub_df)
return results
# Example usage
# df has columns: event_id, variant, intent_class, clicked
# results = analyze_ab_test(df, segment_col=\'intent_class\')
# Inspect results[\'overall\'] for primary; segments for
differential effects.