Fairness in A/B Testing

Pre-registering a heterogeneous treatment effect analysis plan before launching an experiment. This involves defining key sensitive segments (e.g., based on geography, device type, or user tenure) and specifying that the primary success metric must be statistically significant and directionally consistent within each segment.

• Avoids the pitfall of a 'lifted' aggregate metric masking a negative experience for a minority group.
• Requires a stratified sampling strategy to ensure sufficient statistical power within each pre-defined segment.

Implementing non-primary metrics specifically designed to detect harm. An equity guardrail monitors a protected group's experience and triggers an automatic alert or experiment shutdown if a statistically significant negative deviation occurs.

• Example: A new recommendation model increases overall click-through rate but a guardrail detects a 5% drop in content diversity served to a specific demographic.
• These are distinct from standard business guardrails (e.g., latency, revenue) and focus purely on disaggregated impact.

A gradual exposure strategy that starts with a small, diverse subset of the population before full-scale rollout. A fairness-aware ramp-up explicitly validates segment-level metrics at each stage (e.g., 1%, 10%, 50% traffic).

• If a disparate impact signal is detected at 10% traffic, the test is halted for a deep-dive analysis before it affects the broader user base.
• This limits the 'blast radius' of an unintentionally biased model.

A mandatory retrospective analysis conducted after an experiment concludes, regardless of the aggregate result. This audit uses fairness metrics like Equalized Odds and Demographic Parity on the collected experimental data.

• Quantifies the fairness-utility trade-off observed during the test.
• Generates a standardized Model Card entry documenting the model's performance across segments, ensuring transparent handoff from experimentation to production deployment.

Verifying that the causal mechanism of the treatment is stable across groups. This involves testing if the relationship between an intermediate variable (e.g., a new UI element's visibility) and the final outcome is consistent.

• A classic violation is Simpson's Paradox, where a trend appears in several groups of data but disappears or reverses when the groups are combined.
• This check ensures the treatment's theory of change is universally valid, not just an artifact of a specific majority group's behavior.

Maintaining a persistent, randomly assigned control group that is withheld from the treatment for an extended period to measure long-term, equilibrium effects. This is critical for detecting feedback loop bias.

• A short-term A/B test might show a new pricing algorithm increases revenue, but a long-term holdout reveals it causes customer churn in a specific segment after three months.
• This design captures the systemic, delayed effects that standard short-duration experiments miss.