Peeking Problem

The peeking problem arises when an experimenter continuously monitors a test and stops it the moment a p-value crosses a significance threshold (e.g., p < 0.05). Because p-values fluctuate randomly during an experiment, a test that is stopped early upon seeing a 'significant' result has a much higher probability of being a false positive than the nominal 5% alpha level suggests. This violates the fixed-horizon assumption of classical frequentist inference.

Repeated peeking can inflate the Type I error rate far beyond the intended 5% threshold. For example, peeking at data every single day can raise the actual false positive probability to over 25% by the time a decision is made. This occurs because each peek represents an additional opportunity to randomly cross the significance boundary, a phenomenon known as alpha spending or the problem of multiple comparisons over time.

To safely conduct interim analyses, experimenters must use sequential testing frameworks, such as the Sequential Probability Ratio Test (SPRT) or group sequential designs. These methods adjust the significance thresholds at each interim look to maintain the overall desired Type I error rate. Unlike naive peeking, sequential testing defines a stopping boundary that accounts for the number of previous analyses, allowing for valid early decisions.

The peeking problem is a specific limitation of frequentist inference. Bayesian inference does not suffer from this issue because the posterior probability distribution of the treatment effect is updated continuously as data arrives. A Bayesian analysis can be monitored at any time without inflating error rates, as the decision to stop is based on the current probability that one variant is better, not on a fixed significance threshold.

A common pitfall is the 'wait for significance' approach, where a product manager runs an underpowered test and simply lets it run until the p-value drops below 0.05. This guarantees that any reported effect is biased upwards and likely a false positive. The correct procedure is to pre-register a fixed sample size based on a power analysis for a specific minimum detectable effect and only analyze results once that sample size is reached.

Peeking is particularly dangerous in underpowered tests. If a test lacks the statistical power to detect a realistic effect size within a reasonable timeframe, stakeholders are incentivized to peek and stop early. This creates a vicious cycle where the only 'significant' results that are ever observed are the ones that are massively overestimated due to random noise, leading to a 'winner's curse' in the experimentation program.

A statistical framework designed as the direct antidote to the Peeking Problem. Unlike fixed-horizon tests, sequential analysis allows for continuous or intermittent data evaluation without inflating the Type I Error rate.

• Adjusts significance thresholds dynamically after each interim look
• Uses alpha-spending functions to allocate the total allowable false positive rate across multiple analyses
• Enables early stopping for both efficacy and futility without compromising statistical validity
• Common implementations include the Sequential Probability Ratio Test (SPRT) and Group Sequential Designs

The probability of observing a test statistic at least as extreme as the one calculated, assuming the null hypothesis is true. The Peeking Problem directly exploits a fundamental misunderstanding of this metric.

• A p-value of 0.03 does not mean there is a 97% chance the variant is better
• Under repeated interim peeking, the probability of observing p < 0.05 at some point drifts far above the nominal 5% threshold
• The p-value fluctuates wildly in the early stages of an experiment due to high variance in small samples
• Proper interpretation requires a pre-registered analysis plan and a fixed sample size

A false positive error that occurs when the null hypothesis is incorrectly rejected. The Peeking Problem is the single most common procedural cause of inflated Type I Error rates in online experimentation.

• The nominal rate is typically set at α = 0.05 (5%)
• Repeatedly testing accumulating data with a fixed α can inflate the actual error rate to 20-30% or higher
• This inflation occurs because each interim peek represents an independent opportunity to commit a Type I Error
• Controlling this error is the primary objective of sequential testing methodologies and alpha-spending functions

The probability that a statistical test will correctly reject a false null hypothesis, representing the experiment's sensitivity to detect a true effect. The Peeking Problem creates a dangerous illusion of high power.

• Power is formally defined as 1 - β, where β is the probability of a Type II Error
• Peeking and stopping when significant artificially truncates the sampling distribution, making underpowered studies appear conclusive
• A standard target is 80% power at a specified Minimum Detectable Effect
• Proper power analysis must be conducted before the experiment launches, not dynamically during data collection

The expected proportion of rejected null hypotheses that are actually true. In large-scale experimentation platforms, the Peeking Problem contributes directly to an elevated False Discovery Rate.

• Distinct from the Family-Wise Error Rate (FWER) controlled by Bonferroni correction
• The Benjamini-Hochberg procedure is a common method for controlling FDR in multiple testing scenarios
• When product teams peek at dozens of metrics simultaneously and stop on the first significant result, the FDR can approach 100%
• Mitigation requires both statistical corrections and organizational discipline around pre-registration

The smallest statistically significant improvement or degradation that an experiment is designed to reliably detect. The Peeking Problem allows experimenters to claim significance for effects far smaller than the pre-planned MDE.

• A critical input for power analysis that determines required sample size and duration
• If an experiment is designed for a 2% lift MDE, stopping early when observing a 0.5% lift with p < 0.05 is statistically invalid
• The observed effect at an interim peek is a biased estimator of the true effect due to truncation bias
• Winners stopped early will systematically regress to the mean when rolled out to the full population