Confidence Interval

The coverage probability is the long-run frequency with which the interval captures the true parameter. A 95% confidence interval does not mean there is a 95% chance the parameter lies within the calculated bounds; rather, if the experiment were repeated infinitely, 95% of the constructed intervals would contain the true value. The parameter is fixed, and the interval is random.

The width of a confidence interval is inversely proportional to the square root of the sample size. To halve the margin of error, you must quadruple the sample size. This relationship is critical for power analysis: - Small samples yield wide, imprecise intervals - Large samples yield narrow intervals but may detect trivial effects - Diminishing returns set in as sample size grows

The confidence level determines the critical value multiplier. Common pairings include: - 90% confidence → z = 1.645 - 95% confidence → z = 1.96 - 99% confidence → z = 2.576 Higher confidence levels produce wider intervals, reflecting greater certainty that the interval captures the parameter. The trade-off is reduced precision.

Confidence intervals rely on underlying assumptions that must be verified: - Normality: The sampling distribution of the estimator is approximately normal (often satisfied via the Central Limit Theorem) - Independence: Observations are independent; violations occur with clustered or repeated-measures data - Constant variance: Homoscedasticity is assumed in classical intervals Violating these assumptions can lead to intervals with incorrect coverage.

A confidence interval provides a direct duality with a two-sided hypothesis test. If a 95% confidence interval for a difference in means excludes zero, the null hypothesis of no difference is rejected at the α = 0.05 significance level. Conversely, if the interval contains zero, the result is not statistically significant. This makes intervals more informative than p-values alone.

When parametric assumptions are violated, bootstrap methods construct intervals by resampling the observed data with replacement. Common approaches include: - Percentile bootstrap: Takes empirical quantiles of the bootstrap distribution - Bias-corrected and accelerated (BCa): Adjusts for skewness and bias - Studentized bootstrap: Accounts for variance instability These non-parametric intervals are robust to distributional assumptions.

The probability of observing a test statistic at least as extreme as the one calculated, assuming the null hypothesis is true. It is a threshold for determining statistical significance, not the probability that the null hypothesis is false.

• Common Threshold: A p-value < 0.05 is a conventional standard for rejecting the null hypothesis.
• Misinterpretation Risk: A p-value does not measure the magnitude or practical importance of an effect.
• Relation to Confidence Intervals: A 95% confidence interval that does not contain the null value corresponds to a p-value < 0.05.

The probability that a test will correctly reject a false null hypothesis, representing the experiment's sensitivity to detect a true effect if one exists.

• Power Analysis: A pre-experiment calculation to determine the required sample size for a given Minimum Detectable Effect (MDE).
• Trade-off: Increasing power typically requires a larger sample size or a longer experiment duration.
• Standard Target: Experiments are commonly designed to achieve 80% statistical power.

A quantitative measure of the magnitude of the difference between two groups, providing a standardized assessment of practical significance independent of sample size.

• Cohen's d: A common measure for the standardized difference between two means.
• Practical vs. Statistical: A result can be statistically significant but have a negligible effect size, meaning the business impact is trivial.
• Use Case: Essential for comparing results across experiments with different metrics or scales.

A reinforcement learning approach that dynamically allocates more traffic to better-performing variants in real-time, minimizing the opportunity cost of exploration.

• Epsilon-Greedy: A simple algorithm that exploits the best-known action most of the time but selects a random action with probability epsilon.
• Thompson Sampling: A Bayesian approach that selects variants based on their probability of being the best.
• Contrast with A/B Tests: Unlike fixed-horizon tests, bandits continuously shift traffic to the winner, optimizing for cumulative reward during the experiment.

The two fundamental error classes in hypothesis testing that represent a critical trade-off.

• Type I Error (False Positive): Incorrectly rejecting a true null hypothesis, controlled by the significance level (alpha).
• Type II Error (False Negative): Incorrectly retaining a false null hypothesis, inversely related to statistical power.
• The Trade-off: Decreasing the risk of a Type I error by lowering alpha increases the risk of a Type II error, requiring a larger sample size to maintain power.

The statistical bias introduced when an experimenter repeatedly checks interim results and stops an experiment early upon seeing a significant p-value, dramatically inflating the false positive rate.

• Intuition: Continuous monitoring makes it almost certain that a p-value will randomly dip below 0.05 at some point, even with no true effect.
• Mitigation: Use sequential testing methods or pre-register a fixed sample size and duration to avoid invalidating results.
• Impact: Can lead to deploying 'winning' variants that actually provide no improvement or even cause harm.