A/B Testing

The foundational mechanism that eliminates selection bias. Users must be randomly assigned to the control (A) or variant (B) group using a deterministic hashing algorithm or a cryptographically secure pseudo-random number generator. Without true randomization, confounding variables—such as time of day, device type, or user cohort—can systematically skew results. Proper assignment ensures that the only expected difference between groups is the variable being tested.

A result is statistically significant when the observed difference is unlikely to have occurred by random chance, typically measured by a p-value < 0.05. However, significance alone is insufficient. A pre-test power analysis must determine the required sample size to detect a meaningful effect. Key inputs include:

• Minimum Detectable Effect (MDE): The smallest lift worth capturing
• Baseline conversion rate
• Desired statistical power (standard: 80%) Running a test without adequate power risks a false negative, concluding no effect exists when one actually does.

A valid A/B test changes exactly one independent variable between the control and variant. Testing multiple changes simultaneously—such as a new headline, button color, and image—creates a multivariate problem where the driver of any observed effect is unidentifiable. This principle enforces causal attribution. If multiple elements must be tested concurrently, a multivariate test (MVT) with a full factorial design is required, demanding significantly larger sample sizes.

The test duration and sample size must be calculated and locked before the experiment begins. Continuously monitoring results and stopping early upon seeing a significant p-value—a practice known as data peeking—dramatically inflates the false positive rate, often above 25%. Sequential testing methods, such as the Sequential Probability Ratio Test (SPRT), exist for continuous monitoring but require adjusted significance thresholds. The standard approach commits to a fixed horizon and evaluates only once.

A global result can mask heterogeneous treatment effects across subpopulations. Post-test segmentation analysis examines the treatment effect within distinct cohorts, such as new vs. returning users, mobile vs. desktop, or geographic regions. A Simpson's Paradox scenario can occur where a variant appears superior overall but is inferior within every segment. Analyzing interaction effects ensures a winning variant is not deployed to a segment where it causes harm.

A result can be statistically significant yet practically meaningless. A 0.01% lift in click-through rate with a p-value of 0.001 is a true effect but may not justify the engineering cost of deployment. The Minimum Detectable Effect (MDE) defined during power analysis serves as the bridge: if the observed effect is smaller than the MDE, the test is inconclusive regardless of the p-value. Decision-making must weigh effect size, confidence interval width, and business impact.