Online Controlled Experiment

The cornerstone of causal inference. Users must be randomly assigned to control or treatment groups using a deterministic hashing algorithm on a stable user ID. This ensures that, on average, all known and unknown confounding covariates are balanced. A violation of the Stable Unit Treatment Value Assumption (SUTVA), known as the Interference Effect, occurs when the treatment of one user influences the outcome of another, breaking independence.

An experiment must be adequately powered to detect a meaningful change. A Power Analysis calculates the required sample size based on:

• Minimum Detectable Effect (MDE): The smallest lift you care about.
• Statistical Power: Typically set to 80%, representing the probability of correctly rejecting a false null hypothesis.
• Significance Level (Alpha): Usually 5%. Insufficient power leads to Type II Errors (false negatives), missing real improvements.

Experiments must not sacrifice long-term health for short-term gains. Guardrail Metrics are secondary organizational checks that protect the business from unintended harm. Key trustworthiness checks include:

• Sample Ratio Mismatch (SRM): Verifying the actual traffic split matches the design to detect buggy randomization.
• Data Leakage: Ensuring no future information contaminates the training data.
• Peeking Problem: Avoiding the inflation of false positives by not stopping tests early based on interim significance.

A structured statistical framework is required to draw conclusions. The process begins by assuming the Null Hypothesis (no difference exists). A P-Value is calculated to measure the probability of observing the data if the null were true. If the p-value falls below the significance threshold, the null is rejected. To control for multiplicity when testing many metrics, adjustments like the Bonferroni Correction or False Discovery Rate (FDR) control are essential to prevent Type I Errors (false positives).

Standard A/B tests measure short-term impact but miss novelty effects or long-term user fatigue. A Holdout Group is a stable, permanent subset of users excluded from all experimental treatments. This group serves as a global baseline to measure the aggregate, cumulative causal impact of all model changes over months or years, preventing the tragedy of the commons where individually winning tests degrade the overall ecosystem.

Even with randomization, finite samples can be imbalanced. Stratified Sampling ensures critical segments are proportionally represented. During analysis, Covariate Shift must be monitored—if the input feature distribution diverges from the training set, the model's performance metrics become unreliable. Techniques like propensity score matching can adjust for residual imbalances, but they cannot replace proper randomization.

The probability that a statistical test will correctly reject a false null hypothesis. It represents the experiment's sensitivity to detect a true effect if one exists.

• Standard target is 80% (0.80), meaning a 20% chance of a Type II Error (false negative)
• Directly influenced by sample size, effect size, and significance level
• Insufficient power is the primary reason experiments fail to detect genuine improvements
• Calculated during power analysis before launching an experiment

The probability of observing a test statistic at least as extreme as the one calculated, assuming the null hypothesis is true. It is a measure of surprise, not a measure of effect magnitude.

• A p-value of 0.03 means: if there were truly no effect, you'd see results this extreme only 3% of the time
• The common threshold for significance is α = 0.05
• Does not measure the probability that the null hypothesis is true
• Highly susceptible to the peeking problem if interim checks are performed

A reinforcement learning approach that dynamically allocates more traffic to better-performing variants in real-time, minimizing the opportunity cost of exploration compared to traditional fixed-horizon tests.

• Epsilon-Greedy: exploits the best variant but explores randomly with probability ε
• Thompson Sampling: uses Bayesian inference to sample from posterior distributions
• Upper Confidence Bound (UCB): selects the arm with the highest upper bound on expected reward
• Reduces cumulative regret but complicates causal impact measurement for long-term metrics

A secondary organizational metric monitored during an experiment to ensure that a new model or feature variant does not cause unintended harm to the business.

• Trustworthy metrics: verify experimental integrity (e.g., Sample Ratio Mismatch checks)
• Business metrics: protect revenue and user experience (e.g., latency, crash rate, gross merchandise volume)
• Trigger automatic alerts or experiment shutdown if degradation exceeds a predefined threshold
• Essential for fairness-aware personalization to detect disparate impact across segments

A violation of the Stable Unit Treatment Value Assumption (SUTVA) where the treatment applied to one experimental unit influences the outcome of another.

• Common in two-sided marketplaces: treating a driver affects the experience of riders
• Occurs in social networks: a user's new feature changes their friends' behavior
• Mitigated by cluster randomization (randomizing at the city or network level)
• Ignoring interference leads to biased effect size estimates and incorrect conclusions

A long-term, stable subset of users who are permanently excluded from any experimental treatments to serve as a global baseline.

• Measures the aggregate, cumulative impact of all model changes over months or years
• Prevents innovation theater: ensures short-term A/B wins translate to genuine long-term value
• Typically 1-5% of total traffic, kept in a pristine, untreated state
• Critical for validating North Star Metric trends against a true control