Stratified Sampling

Stratified sampling divides the experimental population into distinct, non-overlapping strata based on pre-experiment covariates known to correlate with the outcome metric. Randomization is then performed independently within each stratum. This guarantees that the treatment and control groups have near-identical distributions of these critical features, eliminating the random imbalances that plague simple random assignment. Common stratification variables include country, device type, acquisition channel, and historical user activity tiers.

The statistical power of stratified sampling derives from removing the between-strata variance from the treatment effect estimator. The total variance of the metric is decomposed into within-stratum variance and between-stratum variance. By forcing balance across strata, the between-stratum component is eliminated from the comparison. The resulting variance reduction is proportional to the predictive power of the stratification variables. Even a modestly correlated covariate can yield a 10-30% reduction in the required sample size.

Even when pre-stratification is not implemented at randomization time, analysts can apply post-stratification during the analysis phase. This technique computes the treatment effect separately within each observed stratum and then calculates a weighted average based on the stratum sizes in the full population. While it does not achieve the full efficiency of pre-stratified randomization, it corrects for observed imbalances and provides a consistent estimator. This is a critical safeguard when a Sample Ratio Mismatch is detected.

Stratified sampling and Controlled-experiment Using Pre-Experiment Data (CUPED) are complementary variance reduction techniques. Stratification operates on discrete covariates at the assignment stage, ensuring balance. CUPED uses continuous pre-experiment metrics as a covariate in a linear regression adjustment during analysis. The most robust experimentation platforms combine both: stratifying on categorical variables like platform and applying CUPED on continuous pre-period metrics like historical spend. This dual approach maximizes sensitivity.

In production systems, stratification is implemented via a hash-based assignment function. The user ID and stratum identifier are concatenated and passed through a consistent hashing algorithm. This ensures deterministic, repeatable assignment that is stable across restarts. Key considerations include:

• Strata granularity: Too many strata with small populations can lead to empty cells.
• Adaptive stratification: Strata definitions must be updated as user populations shift.
• Interaction effects: Stratification does not correct for interference between units.

The primary risk is over-stratification, where dividing the population into too many fine-grained strata creates sparsity and invalidates the asymptotic assumptions of the variance estimator. Another failure mode is stratifying on a variable that is a post-treatment mediator rather than a pre-experiment covariate, which introduces collider bias and distorts the treatment effect estimate. Always verify that stratification variables are immutable attributes recorded before the user's exposure to the experiment.

A specific type of data distribution change where the input feature distribution differs between the training and inference environments. In A/B testing, unaddressed covariate shift can invalidate results if the treatment and control groups are not balanced on critical covariates. Stratified sampling is a primary defense against this, ensuring that the distribution of key segments—such as mobile vs. desktop users or new vs. returning visitors—remains consistent across experimental arms. Techniques like propensity score matching are often used post-hoc to correct for covariate shift when stratification wasn't applied during randomization.

A violation of the Stable Unit Treatment Value Assumption (SUTVA) where the treatment applied to one experimental unit influences the outcome of another. This is common in two-sided marketplaces (e.g., ride-sharing) or social networks. Stratified sampling can mitigate interference by defining strata based on geographic clusters or network communities, ensuring that entire clusters are assigned to the same experimental arm. This prevents a treated user's behavior from contaminating a control user's experience, preserving the independence required for valid causal inference.

A statistical phenomenon where a trend appears in several different groups of data but disappears or reverses when these groups are aggregated. For example, a new recommendation model might appear to increase overall conversion rate, but when stratified by traffic source, it actually decreases conversion for both organic and paid users individually. Stratified sampling helps prevent this paradox by ensuring that segment-level analysis is planned from the start. By pre-defining strata on variables like device type or geography, experimenters can detect and diagnose these reversals before drawing incorrect conclusions.

A diagnostic check that verifies if the observed traffic split between control and treatment groups matches the intended randomization ratio. A mismatch is a primary guardrail for detecting bugs in the experimentation infrastructure, such as bot traffic, faulty hash functions, or logging errors. When stratified sampling is used, the SRM check must be performed within each stratum in addition to the overall population. A correct overall ratio can mask severe imbalances within individual strata, which would invalidate the variance reduction benefits that stratification is designed to provide.

The smallest statistically significant improvement or degradation that an experiment is designed to reliably detect. Stratified sampling directly reduces the residual variance in the experimental metric, which in turn reduces the Minimum Detectable Effect (MDE) for a given sample size. By removing the variance explained by the stratification covariates—such as customer lifetime value tier or browser type—the experiment becomes more sensitive. This allows teams to ship smaller, incremental improvements with confidence, accelerating the velocity of model iteration.

A conservative statistical adjustment applied when testing multiple hypotheses simultaneously to control the family-wise error rate. In stratified experiments, analysts often examine treatment effects within each stratum (e.g., 'Did the new model work for high-value users?'). This segmented analysis multiplies the number of hypothesis tests, dramatically inflating the probability of a false positive. The Bonferroni correction divides the significance threshold (alpha) by the number of comparisons, ensuring that segment-level insights are statistically rigorous rather than artifacts of multiple testing.