Frequentist Inference

Frequentist inference defines probability strictly as the limiting frequency of an event occurring over an infinite number of repeated trials. Unlike Bayesian methods, it does not quantify subjective belief or prior knowledge. A p-value of 0.03 means that if the experiment were repeated infinitely under the null hypothesis, a result as extreme as the observed one would occur only 3% of the time. This framework treats the unknown population parameter as a fixed, non-random constant, and the data as the random variable.

The core mechanism involves setting up a null hypothesis (H₀) representing the status quo and an alternative hypothesis (H₁). Key components include:

• Test Statistic: A single number (e.g., t-statistic) summarizing the sample data.
• Rejection Region: The set of values for which the null hypothesis is rejected.
• Type I Error (α): The probability of rejecting a true null, typically fixed at 0.05.
• Type II Error (β): The probability of failing to reject a false null. The decision is binary: reject H₀ or fail to reject H₀, with no probability statement about the hypothesis itself.

A 95% confidence interval does not mean there is a 95% probability that the true parameter lies within the calculated bounds. The correct frequentist interpretation is procedural: if the experiment were repeated many times, 95% of the constructed intervals would capture the fixed true parameter. The interval itself is the random variable. This is a critical distinction from credible intervals in Bayesian inference, which do provide a direct probability statement about the parameter.

The primary method for estimating population parameters in frequentist statistics. Maximum Likelihood Estimation (MLE) selects the parameter value that maximizes the likelihood function—the probability of observing the actual sample data given the parameter. Key properties:

• Consistency: The estimator converges to the true value as sample size increases.
• Asymptotic Normality: The sampling distribution approaches a normal distribution.
• Efficiency: Achieves the lowest possible variance among unbiased estimators. MLE underpins logistic regression, linear models, and many A/B testing calculations.

Frequentist inference heavily depends on the Central Limit Theorem (CLT), which states that the sampling distribution of the mean approaches a normal distribution as sample size increases, regardless of the underlying population distribution. This justifies the use of z-tests and t-tests in A/B testing even when raw metrics like revenue are highly skewed. The CLT enables the construction of confidence intervals and the calculation of p-values without requiring parametric assumptions about the data-generating process.

When testing many variants or metrics simultaneously, the probability of a false positive inflates dramatically. Frequentist inference addresses this through adjustments:

• Bonferroni Correction: Divides the significance threshold α by the number of tests.
• Holm-Bonferroni Method: A sequentially rejective, less conservative procedure.
• False Discovery Rate (FDR): Controls the expected proportion of false rejections. These corrections are essential in experimentation platforms where thousands of metrics are evaluated concurrently to maintain statistical rigor.

The probability of observing a test statistic at least as extreme as the one calculated, assuming the null hypothesis is true. It is the primary decision-making threshold in frequentist inference.

• Interpretation: A low p-value (typically < 0.05) suggests the observed data is unlikely under the null hypothesis.
• Common Misconception: It does not represent the probability that the null hypothesis is true or that results are due to chance.
• Usage: Compared against a pre-defined significance level (alpha) to reject or fail to reject the null.

A range of values derived from sample data that is likely to contain the true population parameter with a specified confidence level (e.g., 95%).

• Frequentist Interpretation: If the experiment were repeated many times, 95% of the calculated intervals would capture the true parameter.
• Practical Use: Provides a measure of precision and uncertainty around a point estimate like the lift in conversion rate.
• Decision Rule: If the interval for a metric delta excludes zero, the result is statistically significant.

The two fundamental error classes in hypothesis testing that frequentist inference seeks to control.

• Type I Error (False Positive): Incorrectly rejecting a true null hypothesis. Controlled by the significance level (alpha).
• Type II Error (False Negative): Failing to reject a false null hypothesis. Controlled by statistical power (1 - beta).
• Trade-off: Decreasing alpha reduces Type I errors but increases the risk of Type II errors unless sample size is increased.

A pre-experiment calculation to determine the minimum sample size required to detect a specific effect with a given level of confidence.

• Key Inputs: Minimum Detectable Effect (MDE) , significance level (alpha), and desired statistical power (1 - beta).
• Purpose: Prevents underpowered experiments that waste traffic and fail to detect real improvements.
• Outcome: Provides the estimated duration an A/B test must run before valid conclusions can be drawn.

A conservative adjustment method to control the family-wise error rate when testing multiple hypotheses simultaneously.

• Mechanism: Divides the desired alpha level by the number of tests performed (e.g., 0.05 / 20 tests = 0.0025).
• Context: Essential in large-scale experimentation platforms where evaluating hundreds of metrics inflates the probability of false positives.
• Trade-off: Reduces statistical power, making it harder to detect true effects. Alternatives like False Discovery Rate (FDR) control are often preferred.

The statistical bias introduced when experimenters repeatedly check interim results and stop an experiment early upon seeing a significant p-value.

• Consequence: Dramatically inflates the Type I error rate far beyond the nominal alpha level.
• Frequentist Solution: Use sequential testing methods with adjusted stopping boundaries or pre-commit to a fixed sample size.
• Best Practice: Avoid making decisions until the pre-calculated sample size is reached unless using a properly designed sequential analysis framework.