P-Value

The most pervasive error is interpreting the p-value as the probability that the null hypothesis is true. A p-value of 0.03 does not mean there is a 97% chance the treatment effect is real. It means: Assuming the null hypothesis is true and there is no effect, there is a 3% probability of observing data as extreme as the collected sample. The p-value conditions on the null hypothesis; it does not quantify the probability of the hypothesis itself. Bayesian methods are required to directly assign probabilities to hypotheses.

A tiny p-value does not imply a meaningful business impact. With a sufficiently large sample size—common in e-commerce platforms processing millions of users—even a negligible effect size of a 0.01% lift in click-through rate can yield a statistically significant p-value. Always report and evaluate the confidence interval of the metric delta alongside the p-value. A result is only actionable if the magnitude of the improvement justifies the engineering cost of deployment.

The dichotomy of 'significant' (p < 0.05) versus 'not significant' (p > 0.05) is an arbitrary convention popularized by R.A. Fisher. There is no statistical discontinuity at this threshold. A p-value of 0.049 and 0.051 are essentially identical in their evidential weight. The American Statistical Association explicitly warns against binary decision-making based solely on p-value thresholds. Treat p-values as a continuous measure of surprise against the null model, not a litmus test for publication.

Failing to reject the null hypothesis (p > 0.05) is frequently misinterpreted as proving the null hypothesis is true. This is a fundamental logical error. A non-significant result simply means the data are insufficient to detect an effect, not that no effect exists. This often occurs in underpowered experiments with small sample sizes. To assert equivalence or the absence of a meaningful effect, you must use a two one-sided test (TOST) procedure, not a standard null hypothesis significance test.

P-hacking refers to the conscious or unconscious manipulation of data analysis until a significant p-value is obtained. Common practices include: collecting more data after a non-significant result (peeking problem), testing multiple dependent variables and reporting only the significant ones, or selectively excluding outliers post-hoc. Without a pre-registered analysis plan and corrections like the Bonferroni correction or False Discovery Rate control, the reported p-value is severely deflated and the Type I error rate is inflated far beyond the nominal 5%.

A significant p-value in a single experiment does not guarantee the result will replicate. Simulation studies show that for a true effect size just barely detectable at 80% power, a p-value of 0.05 corresponds to a replication probability of only approximately 50%. The p-value captures the evidence in a single sample against a specific null model; it does not encode the underlying statistical power or the prior probability of the hypothesis, both of which are critical for predicting whether the effect will hold in a future experiment.