Confidence Interval

The point estimate is the single best guess for the population parameter, calculated from the sample data. In an A/B test, this is typically the observed difference in conversion rates between the treatment and control groups.

• Example: If Variant A has a 5.2% conversion rate and Variant B has a 4.8%, the point estimate for the lift is 0.4 percentage points.
• It serves as the center of the confidence interval but does not convey uncertainty on its own.

The margin of error is the radius of the confidence interval, representing the maximum expected difference between the point estimate and the true population parameter. It is calculated using the standard error of the estimate and a critical value from a probability distribution (e.g., Z or t-distribution).

• Formula: Margin of Error = Critical Value * Standard Error.
• A larger sample size reduces the standard error, resulting in a tighter margin of error and a more precise interval.

The confidence level (e.g., 95%, 99%) expresses the long-run frequency with which the calculated interval would contain the true parameter if the experiment were repeated indefinitely. It is not the probability that a specific interval contains the truth.

• A 95% confidence level implies that 95 out of 100 similarly constructed intervals from repeated sampling would contain the true effect.
• Higher confidence levels (e.g., 99%) produce wider intervals, trading precision for greater certainty.

The standard error measures the variability or precision of the point estimate (like a mean or proportion difference). It is the estimated standard deviation of the sampling distribution of the statistic.

• Key Driver: It is inversely related to the square root of the sample size (SE ≈ σ/√n). Doubling the sample size reduces the standard error by about 30%.
• In A/B testing for proportions, the standard error for the difference is calculated using the pooled variance from both groups.

The critical value is a multiplier derived from a theoretical probability distribution (Z for large samples, t for small samples) corresponding to the chosen confidence level. It defines how many standard errors to extend from the point estimate.

• For a 95% confidence level using a normal approximation, the Z-score is approximately 1.96.
• The t-score is used with smaller samples and has a larger value, creating a wider interval to account for additional uncertainty in estimating the population standard deviation.

The final component is the interpretive rule linking the interval to a business decision. The interval's relationship to a null value (often zero, meaning 'no effect') determines statistical significance.

• Rule: If a 95% CI for a lift excludes zero, the result is statistically significant at the 5% level.
• Decision Boundary: The interval provides a range of plausible values for the true effect. If the entire interval lies above a minimum practical significance threshold, it supports a confident launch decision.

Statistical significance is a determination that an observed difference between experimental groups (e.g., a lift in a key metric) is unlikely to have occurred due to random chance alone. It is formally assessed by comparing a calculated p-value to a pre-defined significance level (alpha), commonly set at 0.05. A result is deemed statistically significant if the p-value is less than alpha, providing evidence to reject the null hypothesis of no effect. It is crucial to note that statistical significance does not imply practical importance; a tiny effect can be significant with a large enough sample size.

A p-value is the probability, assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one calculated from the sample data. It quantifies the strength of evidence against the null hypothesis.

• A low p-value (e.g., < 0.05) suggests the observed data is inconsistent with the null hypothesis.
• It is not the probability that the null hypothesis is true, nor the probability that the alternative hypothesis is false.
• Misinterpretation of p-values is a common source of error in experiment analysis. They must be considered alongside effect size and confidence intervals.

Statistical power is the probability that a test will correctly reject a false null hypothesis (i.e., detect a true effect). Power is influenced by:

• Sample Size: Larger samples increase power.
• Effect Size: Larger true effects are easier to detect.
• Significance Level (Alpha): A higher alpha (e.g., 0.10) increases power but also the false positive rate.

The Minimum Detectable Effect (MDE) is the smallest true effect size an experiment is powered to detect, given a fixed sample size, alpha, and desired power (e.g., 80%). Designing an experiment requires specifying an MDE that is both statistically feasible and business-relevant.

A Multi-Armed Bandit (MAB) is a sequential decision-making framework that dynamically allocates traffic between experimental variants. Unlike fixed-horizon A/B tests, MAB algorithms balance:

• Exploration: Gathering data on uncertain variants.
• Exploitation: Favoring the variant currently estimated to be best.

Thompson Sampling is a prominent Bayesian MAB algorithm. For each allocation decision, it:

1. Samples a potential reward value from the posterior probability distribution of each variant.
2. Selects the variant with the highest sampled value. This approach naturally converges traffic to the optimal variant while minimizing regret during the learning phase.

Causal inference is the discipline of drawing conclusions about cause-and-effect relationships from data. While randomized controlled trials (A/B tests) are the gold standard, causal inference provides methods for observational settings.

The Average Treatment Effect (ATE) is the core target of estimation: the average difference in outcomes if the entire population received the treatment versus if it received the control. Related methodologies include:

• Propensity Score Matching: Reduces bias by matching treated/control units with similar probabilities of treatment.
• Instrumental Variables: Uses a third variable to isolate causal effects.
• Intent-to-Treat Analysis: Analyzes subjects by their originally assigned group, preserving randomization.

Sequential testing is an experimental design where data is analyzed continuously as it accumulates, allowing for early stopping if results become conclusive. This can reduce the required sample size.

The peeking problem is a major risk: repeatedly checking p-values before a planned sample size is reached inflates the Type I error rate (false positives). Each 'peek' is an additional chance to find a statistically significant result by random chance.

• Solution: Use sequential testing procedures with adjusted significance thresholds (e.g., Alpha Spending Functions) that control the overall error rate despite multiple looks at the data. Standard fixed-horizon tests assume a single analysis at the end.