Simpson's Paradox

The defining characteristic: a statistical relationship present in multiple subgroups disappears or reverses direction when the subgroups are combined into a single aggregate view. For example, a new recommendation model may show a positive lift in conversion rate for both mobile and desktop users separately, but when the data is pooled, the aggregate metric shows a negative lift. This occurs because the confounding variable (device type) influences both the assignment to the variant and the baseline conversion rate.

The paradox is always driven by a lurking or confounding variable that correlates with both the independent variable (treatment assignment) and the dependent variable (outcome metric). In experimentation, this often manifests as:

• Unequal sample sizes across segments
• Different baseline conversion rates between segments
• Imbalanced traffic allocation that skews the aggregate Without identifying and controlling for this confound, the aggregate result is mathematically correct but causally misleading.

In online controlled experiments, the paradox emerges when traffic splits are not uniform across user segments. Consider a test where the treatment group receives 80% mobile users (low-converting) and 20% desktop users (high-converting), while the control group has the inverse proportion. Even if the treatment improves conversion within each device segment, the aggregate metric will favor the control group simply because it contains more high-converting desktop users. This is not a model failure — it is a compositional artifact.

The primary defense is to never rely solely on aggregate metrics. Best practices include:

• Stratified sampling: Ensure treatment and control groups have identical covariate distributions at assignment time
• Segment-level reporting: Always examine treatment effects within key subgroups (device, geography, user tenure) before aggregating
• Weighted averages: Use fixed weights based on the population distribution rather than sample proportions to compute an unbiased overall effect
• Regression adjustment: Include the confounding variable as a covariate in a linear model to isolate the treatment effect

The paradox is named after Edward H. Simpson, who described the phenomenon in a 1951 paper, though Karl Pearson and Udny Yule had observed similar effects decades earlier. A famous real-world example is the UC Berkeley gender bias case (1973), where aggregate admission data appeared to show bias against women, but a department-by-department analysis revealed a slight bias in favor of women. The reversal occurred because women applied disproportionately to more competitive departments with lower overall admission rates.

Simpson's Paradox is closely related to Berkson's Paradox, another statistical phenomenon where conditioning on a common effect creates a spurious negative correlation between two independent causes. While Simpson's involves aggregation across subgroups, Berkson's involves selection bias from conditioning. Both highlight the danger of drawing causal conclusions from observational data without a clear understanding of the underlying data-generating process and the role of collider variables.

A randomization technique that divides the population into homogeneous subgroups before sampling to ensure the treatment and control groups are balanced on critical covariates. By controlling for the confounding variable that causes the reversal, stratified sampling prevents Simpson's Paradox from manifesting in the first place.

• Mechanism: Pre-segment users by the confounding variable (e.g., device type, traffic source) before random assignment.
• Benefit: Reduces variance and guarantees proportional representation of segments in both control and treatment arms.
• Contrast: Post-hoc segmentation without stratified assignment is where the paradox thrives.

A range of values, derived from sample data, that is likely to contain the true population parameter with a specified probability. When analyzing segmented results, overlapping confidence intervals between subgroups can signal the presence of Simpson's Paradox.

• Interpretation: A 95% CI means that if the experiment were repeated many times, 95% of the intervals would contain the true effect.
• Paradox Detection: If the aggregate CI points in one direction but every segment's CI points in the opposite direction, the aggregation is confounded.
• Key Metric: Always report CIs at both the aggregate and segment levels.

A specific type of data distribution change where the input feature distribution differs between the training and inference environments. In A/B testing, a covariate shift in the underlying user population mix can trigger Simpson's Paradox by changing the relative sizes of subgroups.

• Example: A mobile-heavy user base during a holiday sale versus a desktop-heavy base during weekdays.
• Impact: The aggregate metric shifts not because of the treatment, but because the weighting of segments changed.
• Mitigation: Use propensity score matching or inverse probability weighting to re-balance the covariate distribution.

A violation of the Stable Unit Treatment Value Assumption (SUTVA) where the treatment applied to one experimental unit influences the outcome of another. This creates a hidden confounding structure that can mimic Simpson's Paradox in two-sided marketplaces.

• Mechanism: If treating a power seller changes the experience for buyers in the control group, the groups are no longer independent.
• Paradox Link: The interference creates a latent variable that confounds the relationship between treatment and outcome.
• Detection: Use network-clustered randomization or ego-network analysis to isolate interference.

A time-series analysis methodology developed by Google that constructs a synthetic counterfactual baseline to estimate the causal effect of an intervention. It is invaluable for diagnosing Simpson's Paradox when a randomized control group is unavailable.

• How it works: Builds a Bayesian structural time-series model using control time series to predict the post-intervention period.
• Paradox Application: Run Causal Impact on each segment independently to see if the direction of effect is consistent before aggregating.
• Output: Provides a posterior distribution of the causal effect, not just a point estimate.

A secondary organizational metric monitored during an experiment to ensure that a new model or feature variant does not cause unintended harm. Guardrails are the first line of defense against Simpson's Paradox masking a negative effect in a critical subgroup.

• Segmented Guardrails: Define specific guardrails for high-value segments (e.g., paying users, new users) rather than relying solely on the global average.
• Example: A recommendation model increases aggregate click-through rate but degrades the experience for the top 1% of power users.
• Action: If a segment-level guardrail fails, the experiment is halted regardless of the aggregate North Star Metric.