Causal Inference

The core thought experiment of causal inference: comparing what actually happened to what would have happened under a different condition. For a treated unit, the counterfactual is the outcome it would have experienced had it not received the treatment. The fundamental challenge, the Fundamental Problem of Causal Inference, is that we can never observe both the factual and counterfactual outcomes for the same unit. Causal methods are designed to estimate these unobserved quantities using data from comparable units.

Also known as the Neyman-Rubin Causal Model, this is the dominant mathematical framework for defining causal effects. For each unit i and a binary treatment, it posits two potential outcomes:

• Y_i(1): The outcome if unit i receives the treatment.
• Y_i(0): The outcome if unit i does not receive the treatment.

The individual treatment effect is ITE_i = Y_i(1) - Y_i(0). Since we cannot observe both, we estimate population-level averages like the Average Treatment Effect (ATE) = E[Y(1) - Y(0)]. This framework forces explicit definition of the treatment, the outcome, and the units of analysis.

A critical assumption for estimating causal effects from observational data. Also called conditional independence, it states that treatment assignment is independent of the potential outcomes, given a set of observed covariates X. Formally: (Y(1), Y(0)) ⟂ T | X. This means, within groups of units that are identical on X, the treatment is assigned as if by random. If there are unobserved variables that affect both treatment and outcome (confounders), this assumption is violated, and estimated effects may be biased. Techniques like propensity score matching aim to achieve this balance.

A graphical tool used to encode causal assumptions and identify sources of bias. DAGs consist of:

• Nodes: Representing variables (treatment, outcome, confounders, mediators).
• Directed Edges (→): Representing assumed causal relationships.
• Acyclic Paths: No variable can be its own ancestor.

DAGs allow researchers to visually apply d-separation rules to determine which variables to condition on (or not) to block backdoor paths—non-causal paths that create spurious association. They are essential for formalizing the data-generating process before analysis.

A method for estimating causal effects when unobserved confounding is suspected. An instrumental variable Z must satisfy two key conditions:

1. Relevance: Z is correlated with the treatment variable T.
2. Exclusion Restriction: Z affects the outcome Y only through its effect on T (no direct path).

By using only the variation in T induced by Z, IV methods can isolate the causal effect of T on Y. Common estimators include Two-Stage Least Squares (2SLS). A classic example: using distance to a college as an instrument for education to estimate the effect of education on earnings.

A quasi-experimental design used to estimate causal effects by comparing the change in outcomes over time between a treated group and a non-treated control group. The core parallel trends assumption states that, in the absence of treatment, the difference between the groups' outcomes would have remained constant over time.

The DiD estimator is calculated as: DiD = (Y_treated,post - Y_treated,pre) - (Y_control,post - Y_control,pre) This method differences out time-invariant differences between groups and group-invariant time trends, isolating the treatment effect. It is widely used in policy evaluation and economics.

The core task of causal inference is to answer "what if" questions by estimating what would have happened to a unit (e.g., a user) had they received a different treatment. Key methods include:

• Potential Outcomes Framework: Models each unit's outcome under both treatment and control states.
• Inverse Probability Weighting: Re-weights observed data to simulate a randomized experiment.
• Doubly Robust Estimators: Combine models for the treatment assignment and outcome to provide valid estimates even if one model is misspecified. This is foundational for evaluating the true impact of a new AI model or feature.

In production, randomized A/B tests are not always feasible. Causal methods enable valid inference from observational data by accounting for confounding variables—factors that influence both the treatment assignment and the outcome. Common techniques are:

• Propensity Score Matching: Pairs treated and control units with similar likelihoods of receiving treatment.
• Regression Adjustment: Directly models and controls for confounders in the outcome model.
• Difference-in-Differences: Compares changes over time between a treated group and a control group. This allows for retrospective analysis of model performance or user behavior shifts.

Uplift modeling, or heterogeneous treatment effect estimation, identifies which users are most responsive to a treatment (e.g., a recommendation, discount, or model version). This moves beyond predicting outcomes to predicting the causal effect for each individual. Algorithms include:

• Meta-learners (S-Learner, T-Learner, X-Learner): Use base ML models (like gradient boosting) to estimate conditional average treatment effects.
• Causal Forests: An adaptation of random forests for treatment effect estimation. The output directs personalization strategies, ensuring interventions are deployed only where they have a positive net effect.

This application focuses on learning the underlying causal graph or Directed Acyclic Graph (DAG) from data. It aims to uncover the structure of cause-and-effect relationships between variables. Methods include:

• Constraint-based algorithms (PC, FCI): Use conditional independence tests to infer graph structure.
• Score-based methods: Search over graph space to optimize a goodness-of-fit score with a sparsity penalty.
• Additive Noise Models: Assume functional relationships with non-Gaussian noise to identify directionality. These graphs are used for feature selection, understanding data-generating processes, and informing model design to avoid spurious correlations.

When model performance degrades or data drift is detected, causal inference helps distinguish between:

• Confounding Shifts: Changes in the input distribution (e.g., more premium users).
• Mechanism Shifts: Changes in the true underlying relationship between inputs and output. By formally modeling the data-generating process, engineers can pinpoint whether drift is due to a shift in a causal parent variable (requiring data pipeline fixes) or a breakdown in the learned relationship (requiring model retraining). This moves monitoring from correlation to causation.

Standard A/B tests often measure short-term, direct effects. Causal inference provides tools for assessing more complex impact scenarios:

• Mediation Analysis: Decomposes the total effect of a treatment into direct and indirect effects (e.g., a new UI affects revenue both directly and through increased user engagement).
• Instrumental Variables: Estimates effects when treatment adherence is imperfect or there is unmeasured confounding.
• Spatial/Temporal Interference: Accounts for effects where one user's treatment can influence another user's outcome (e.g., in social networks or marketplace dynamics). This is essential for understanding the full business impact of AI-driven changes.

The Average Treatment Effect is the central quantity estimated in causal inference, representing the average difference in outcomes between a treatment group and a control group across a target population. It answers the question: 'What is the causal effect of the intervention, on average?'

• Calculation: ATE = E[Y(1) - Y(0)], where Y(1) is the potential outcome under treatment and Y(0) is the potential outcome under control.
• Key Assumption: Requires ignorability (no unmeasured confounding) to be interpreted causally.
• Example: In an A/B test for a new recommendation algorithm, the ATE is the average difference in user engagement (e.g., click-through rate) between the group that saw the new algorithm and the group that saw the old one.

Propensity Score Matching is a quasi-experimental method used to estimate causal effects from observational data by reducing selection bias. It matches treated units with untreated units that have a similar probability (propensity) of receiving the treatment based on observed covariates.

• Core Idea: Creates a synthetic control group that is statistically similar to the treatment group on all observed pre-treatment variables.
• Process: 1) Estimate a model (e.g., logistic regression) predicting treatment assignment. 2) Match units (e.g., nearest neighbor, caliper) based on their estimated propensity scores. 3) Compare outcomes within the matched sample.
• Limitation: Can only adjust for observed confounders; hidden bias from unobserved variables remains a threat.

Instrumental Variables is an advanced econometric technique used to estimate causal relationships when controlled experimentation is impossible and unmeasured confounding is suspected. It uses a third variable—the instrument—that affects the treatment but is unrelated to the outcome except through its effect on the treatment.

• Requirements for a Valid Instrument:
  • Relevance: The instrument must be correlated with the treatment variable.
  • Exclusion Restriction: The instrument must affect the outcome only through the treatment (no direct path).
  • Exogeneity: The instrument must be uncorrelated with unobserved confounders.
• Common Example: Using distance to a college as an instrument to estimate the effect of education on earnings, assuming distance affects schooling choice but not earnings directly.

The Potential Outcomes Framework (or Rubin Causal Model) is the dominant mathematical formalism for defining and estimating causal effects. It defines causality in terms of potential, counterfactual states of the world.

• Core Concepts:
  • For each unit i, there exists a potential outcome Y_i(1) if treated and Y_i(0) if not treated.
  • The fundamental problem of causal inference is that we can only observe one of these two potential outcomes for any given unit.
  • Causal effects are defined as comparisons of these potential outcomes (e.g., Y_i(1) - Y_i(0)).
• Role in Experimentation: Randomized controlled trials solve the fundamental problem by ensuring the assignment to treatment is independent of potential outcomes, making the observed average difference an unbiased estimate of the ATE.

Difference-in-Differences is a quasi-experimental design that estimates causal effects by comparing the change in outcomes over time between a group that receives a treatment and a group that does not. It controls for unobserved, time-invariant confounders.

• Calculation: DiD = (Y_treatment,post - Y_treatment,pre) - (Y_control,post - Y_control,pre).
• Key Assumption: The parallel trends assumption—in the absence of treatment, the treatment and control groups would have followed similar trajectories over time.
• Common Use Case: Evaluating the impact of a new policy (e.g., a minimum wage increase in one state) by comparing outcome changes in that state to a similar state without the policy, before and after implementation.

A Causal Graph or Directed Acyclic Graph is a visual and mathematical tool used to encode assumptions about the causal relationships between variables. It is essential for identifying confounding, selecting appropriate adjustment variables, and avoiding bias.

• Elements: Nodes represent variables. Directed edges (arrows) represent assumed causal directions.
• Key Rules: d-separation determines conditional independence relationships implied by the graph.
• Practical Use: Before analyzing data, drawing a DAG forces explicit assumptions about what causes what. It answers: 'What variables must I control for to get an unbiased estimate of the effect of X on Y?'
• Example: A DAG showing that socioeconomic status causes both education level and health outcomes reveals that failing to control for socioeconomic status would confound the observed correlation between education and health.