A/B/n Test

Unlike sequential A/B tests, an A/B/n test evaluates two or more challenger variants against a single control group in a single experimental window. This parallel structure eliminates temporal confounders like seasonality or shifting user demographics that would invalidate comparisons made across sequential tests. The randomization engine must assign users to one of n distinct buckets with equal probability, ensuring each variant receives a statistically valid traffic share without introducing Sample Ratio Mismatch (SRM).

Testing multiple variants inflates the probability of a Type I Error (false positive). If an experimenter runs 20 independent tests at a 0.05 significance level, the chance of at least one false positive is approximately 64%. To maintain statistical rigor, A/B/n frameworks require corrections such as the Bonferroni Correction or control of the False Discovery Rate (FDR). These adjustments re-calibrate the p-value thresholds to ensure the overall probability of a false positive across the entire family of comparisons remains at the desired alpha level.

Adding more variants divides the fixed incoming traffic pool into smaller slices, directly reducing the sample size per variant. This dilution increases the Minimum Detectable Effect (MDE) and extends the time required to achieve adequate Statistical Power. A test with one control and nine treatments allocates only 10% of traffic to each bucket, requiring a substantially larger user base or longer run time to detect the same effect size compared to a simple A/B test. Power Analysis must account for this fragmentation before launch.

A/B/n tests in two-sided marketplaces or social networks risk violating the Stable Unit Treatment Value Assumption (SUTVA). An Interference Effect occurs when a user in variant B interacts with a user in variant C, causing the treatment to bleed across experimental units. For example, a new recommendation algorithm for sellers in one variant may shift inventory visibility for buyers in another. Advanced experimental designs, such as cluster-based or switchback randomization, are required to isolate these network effects.

Traditional fixed-horizon A/B/n tests suffer from opportunity cost: they continue exposing users to inferior variants until the test concludes. A Contextual Multi-Armed Bandit approach dynamically shifts traffic toward the best-performing variants in real-time. Algorithms like Thompson Sampling or Upper Confidence Bound (UCB) balance exploration of new variants with exploitation of current winners, minimizing cumulative regret. This is particularly effective for short-lived content or inventory where learning must be rapid.

With multiple variants, the risk of optimizing for a surface-level metric at the expense of long-term health increases. A/B/n frameworks must monitor Guardrail Metrics—such as latency, crash rates, or gross merchandise volume—to prevent launching a variant that improves click-through rate but degrades system stability. The ultimate decision criterion remains the North Star Metric, the singular KPI representing core user value. A variant must not only beat the control but also pass all guardrail checks to be considered a winner.

The statistical bias introduced when experimenters repeatedly check interim results and stop the test early upon seeing a significant p-value. This practice dramatically inflates the false positive rate far beyond the nominal 5% threshold.

• Checking daily can inflate Type I error to 20-30%
• The p-value fluctuates wildly in early stages due to small sample sizes
• Proper mitigation requires sequential testing frameworks like always-valid p-values or group sequential designs

Always pre-specify the experiment duration and resist the temptation to stop early based on a single significant snapshot.

The smallest practically meaningful improvement that an experiment is powered to detect. Setting the MDE too small requires enormous sample sizes; setting it too large risks missing valuable incremental gains.

• Formula: MDE = (Z_α/2 + Z_β) × σ / √n
• A 1% lift requires roughly 4x the sample of a 2% lift
• MDE should be grounded in business impact, not statistical convenience

For e-commerce personalization, typical MDEs range from 0.5% to 2% relative lift on conversion rate, depending on traffic volume and the cost of implementation.

A diagnostic check verifying that the observed traffic split matches the intended randomization ratio. An SRM is often the first signal of a critical bug in the experimentation infrastructure.

• A chi-squared goodness-of-fit test detects significant deviations
• Common causes: bot traffic, misconfigured routing, residual carryover effects
• An SRM invalidates all other test results regardless of statistical significance

Treat SRM as a hard guardrail metric. If the intended 50/50 split shows 48/52 with p < 0.001, halt the experiment and investigate the randomization pipeline immediately.

A long-term, stable subset of users permanently excluded from all experimental treatments. While individual A/B/n tests measure incremental lift against a concurrent control, holdouts measure the cumulative, aggregate impact of all model changes over time.

• Typically 5-10% of total traffic
• Detects tragedy of the commons where individually positive tests degrade the ecosystem
• Reveals metric drift from model interactions that isolated tests miss

Essential for organizations running dozens of concurrent experiments to ensure the sum of all "statistically significant wins" actually translates to real business growth.

A statistical phenomenon where a trend appears in subgroups but reverses when data is aggregated. In A/B/n testing, this can cause you to ship a variant that appears globally positive but actually harms every user segment.

• Example: Variant B increases conversion for both mobile and desktop users separately, but shows a decrease overall due to a shift in the traffic mix toward lower-converting mobile
• Mitigation requires segmented analysis and understanding of compositional effects
• Always examine results across key covariate dimensions before declaring a winner

This paradox underscores why aggregate metrics alone are insufficient for experimental conclusions.