Stratified Sampling

The population is partitioned into non-overlapping subgroups called strata. The defining characteristic is that members within each stratum are homogeneous with respect to the variable of interest (e.g., user tenure, geographic region, device type). This internal similarity is what reduces variance within strata, making the overall sample more efficient. For example, in an A/B test for a feature, you might create strata based on user subscription tier (Free, Pro, Enterprise) to ensure each tier is proportionally represented in the experiment's traffic split.

Samples are drawn from each stratum, but the method varies:

• Proportional Allocation: Sample size from each stratum is proportional to the stratum's size in the population. This maintains the natural population proportions in the sample.
• Disproportional (Optimal) Allocation: Sample size is allocated to minimize overall variance, often oversampling smaller strata if they have high internal variability. This is used when precise estimates are needed for all subgroups, regardless of their size. In A/B testing, proportional allocation is standard for overall treatment effect estimation, while disproportional may be used for detailed cohort analysis of small but important user segments.

The primary statistical benefit of stratification is a reduction in the sampling error for estimates of the population mean or total. By ensuring all major subgroups are represented, it eliminates the chance of a purely random sample missing a key segment entirely. This leads to narrower confidence intervals and greater statistical power compared to simple random sampling of the same total size. In practical terms, for an A/B test, this means you can detect a smaller minimum detectable effect with the same number of users, or achieve the same precision with a smaller sample size.

In online experimentation, stratified sampling is implemented via deterministic hashing. A user's stable ID (e.g., user_id) and the experiment key are hashed to assign the user to a variant. Crucially, the hashing occurs within each predefined stratum. This guarantees:

• Consistent Assignment: A user always sees the same variant for a given experiment.
• Balanced Covariates: Known confounding variables (strata) are evenly distributed between control and treatment groups.
• Valid Inference: It controls for the influence of the stratification variables, leading to more accurate estimation of the average treatment effect.

It is critical to distinguish stratified sampling from cluster sampling, as they serve opposite purposes.

• Stratified Sampling: Aims for homogeneity within strata and heterogeneity between them. Samples are taken from all strata.
• Cluster Sampling: Aims for heterogeneity within clusters (mini-populations) and homogeneity between them. A random subset of clusters is selected, and all members within chosen clusters are sampled. Stratification is used when a sampling frame for subgroups exists and the goal is precision. Cluster sampling is used for cost or logistical efficiency when the population is naturally grouped (e.g., users by data center).

Even if an experiment uses simple random assignment, post-stratification can be applied during analysis. This involves grouping users into strata after the experiment concludes and re-weighting the results to match the known population proportions. This adjusts for chance imbalances in stratum representation between variants, reducing bias. It is a form of covariate adjustment. The analysis often uses methods like stratified t-tests or regression models that include stratum indicators to compute a weighted average of within-stratum effects, yielding a more precise and less variable estimate of the overall treatment effect.

In live A/B tests, user populations are rarely uniform. Stratified sampling ensures each experimental variant (control/treatment) receives a proportionally representative sample from each key user segment (stratum), such as geographic region, device type, or subscription tier. This prevents skewed results where one variant is accidentally assigned more high-value users, which could bias the primary metric (e.g., conversion rate). By guaranteeing balanced representation, it increases the statistical power of the test and the validity of the average treatment effect calculation.

When constructing datasets to benchmark model performance, naive random sampling can under-represent rare but critical classes. Stratified sampling is used to create a hold-out test set or validation set that mirrors the true class distribution of the production data. For example, in a medical imaging model, it ensures rare diseases are present in the evaluation set. This provides a more accurate estimate of real-world performance and is essential for calculating reliable metrics like precision, recall, and F1-score across all strata.

Drift detection systems monitor the statistical properties of incoming production data versus a reference baseline. Stratified sampling is applied to the live data stream to create manageable, representative samples for daily or hourly analysis. By sampling proportionally from each stratum (e.g., user cohort, product category), the monitoring system can detect covariate shift within specific segments, not just in the aggregate. This enables targeted alerts, such as detecting a performance drop for a new user demographic before it impacts the overall system SLO.

Auditing an AI system for unfair discrimination requires analyzing performance across legally or ethically protected attributes (e.g., gender, age, ethnicity). Stratified sampling is used to construct an evaluation dataset with sufficient sample sizes from each demographic subgroup. This allows for the calculation of disparate impact ratios and subgroup-specific metrics (e.g., accuracy per stratum). Without stratification, minority groups may be absent from the audit sample, rendering the bias assessment incomplete and non-compliant with regulations like the EU AI Act.

During model development, hyperparameter tuning via cross-validation is computationally expensive. Applying stratified sampling within each cross-validation fold ensures that each fold retains the approximate class distribution of the full dataset. This prevents scenarios where a training fold lacks examples of a minority class, which would lead to poor validation scores and unstable tuning results. It leads to more robust hyperparameter selection and reliable estimates of model generalization error.

Evaluating synthetic data generation systems requires verifying that the artificial data preserves the multivariate relationships of the real source data. Stratified sampling is used to create multiple, representative real-data subsets against which synthetic batches are compared. Analysts check if key strata (combinations of sensitive and feature columns) are represented with correct frequencies and correlations in the synthetic output. This stratified assessment is a core component of synthetic data fidelity metrics.

A probability sampling technique where the population is divided into naturally occurring groups (clusters), and entire clusters are randomly selected for inclusion in the sample. This contrasts with stratified sampling, where sub-groups are defined to be homogeneous and samples are taken from each.

• Key Difference: In stratified sampling, all strata are represented. In cluster sampling, only the selected clusters are studied in detail.
• Use Case: More practical and cost-effective when a population is geographically dispersed (e.g., sampling schools within a district rather than individual students nationwide).

A method where sample members are selected from a population at a fixed, periodic interval after a random starting point. For a population of size N and a desired sample size n, the sampling interval k is calculated as N/n.

• Process: 1. Randomly select a starting number between 1 and k. 2. Select every kth element thereafter.
• Consideration: Risk of bias if the population list has a hidden periodic pattern that aligns with the sampling interval.

The application of stratified principles to experimental design, not just sampling. Participants are first divided into strata based on key covariates (e.g., age, usage tier). Within each stratum, they are then randomly assigned to control or treatment groups.

• Purpose: Ensures treatment groups are balanced on known confounding variables, increasing the experiment's internal validity and statistical power.
• Contrasts with: Simple random assignment, which can, by chance, create imbalanced groups on important characteristics.

A non-probability sampling method where the researcher ensures the sample reflects certain characteristics (quotas) of the population. While it resembles stratified sampling in its use of subgroups, it lacks the random selection component.

• Key Limitation: Because selection within quotas is non-random (often via convenience), the sample is not statistically representative, and results cannot be reliably generalized to the population.
• Common Use: Often used in market research and opinion polling where speed and cost override the need for rigorous generalizability.

A survey analysis technique where a sample is re-weighted after data collection to match the known population proportions across strata. This corrects for imbalances that occurred during a simple random or non-stratified sampling process.

• Application: Used to adjust for non-response bias or sampling errors to produce more accurate population estimates.
• Contrast: Unlike stratified sampling (which ensures representation during selection), post-stratification is a correction applied during the analysis phase.

A variant where samples are not allocated proportionally to stratum size. Smaller strata may be oversampled to ensure sufficient data for analysis, and results are later weighted to reflect the true population proportions.

• Primary Reason: To guarantee adequate statistical power for analyzing small but important subgroups (e.g., users of a rare but high-value feature, specific demographic minorities).
• Analytical Requirement: Requires the use of sampling weights in estimation to avoid biasing the overall population estimate.