P-Value

The most common significance level, alpha (α), is set at 0.05. This creates a decision rule:

• p-value ≤ 0.05: Reject the null hypothesis. The observed data is considered statistically significant, suggesting the effect is unlikely due to random chance alone.
• p-value > 0.05: Fail to reject the null hypothesis. The evidence is insufficient to conclude a statistically significant effect.

This threshold is arbitrary but deeply institutionalized. In high-stakes fields like drug trials or aviation AI, a more stringent α = 0.01 or 0.001 may be required.

Misinterpretation is rampant. A p-value is not:

• The probability the null hypothesis is true. (That's a Bayesian posterior probability).
• The probability the observed data occurred by chance. It is the probability of the data or more extreme data, assuming the null is true.
• A measure of effect size or importance. A tiny, practically meaningless effect can be statistically significant (low p-value) with a large enough sample.
• The probability the alternative hypothesis is true.
• A direct measure of replicability. While low p-values suggest replicability, they do not guarantee it.

While the α threshold creates a binary decision, the p-value itself is a continuous measure of evidence. Common interpretations of this gradient include:

• p > 0.10: Little to no evidence against the null.
• 0.05 < p ≤ 0.10: Suggestive but weak evidence.
• 0.01 < p ≤ 0.05: Moderate evidence against the null.
• 0.001 < p ≤ 0.01: Strong evidence.
• p ≤ 0.001: Very strong evidence.

This continuum is why reporting the exact p-value (e.g., p=0.037) is preferred over just "p < 0.05", as it conveys the strength of evidence.

The Type I error rate (false positive rate) is only guaranteed at the α level for a single, pre-planned test. Common practices inflate this error:

• P-Hacking: Trying multiple analyses or data peeks until a p-value < 0.05 is found.
• Multiple Comparisons: Testing many hypotheses (e.g., 20 model metrics) without correction.

If you test 20 independent metrics at α=0.05, the chance of at least one false positive rises to 1 - (0.95)^20 ≈ 64%. Corrections like the Bonferroni correction (α/m) or False Discovery Rate (FDR) control are essential in A/B testing platforms analyzing many guardrail metrics.

In machine learning and A/B testing, the interpretation of p-values must be contextualized by business and engineering realities:

• Guardrail Metrics: A p < 0.05 on a critical guardrail (e.g., latency increase, error rate) may halt a launch, even if the primary metric is significant.
• Sequential Testing: Modern platforms use sequential analysis (e.g., SPRT) that allows for valid peeking and early stopping, dynamically adjusting thresholds as data accumulates.
• Practical vs. Statistical Significance: A model lift with p=0.04 may be statistically significant but have a Minimum Detectable Effect (MDE) below the business's cost-to-implement threshold, making it practically irrelevant.

Over-reliance on p < 0.05 has contributed to a replication crisis in science. Key lessons for AI evaluation:

• Low p-values do not guarantee true effects. Publication bias towards p < 0.05 distorts the literature.
• Power Matters: Underpowered experiments (low statistical power) have a high chance of missing true effects (Type II errors) and, when they do find significance, the effects are often exaggerated.
• Solution Emphasis: The field is shifting towards:
  • Pre-registering analysis plans.
  • Focusing on effect sizes and confidence intervals.
  • Using Bayesian methods that quantify evidence for and against hypotheses.
  • Demanding larger sample sizes for adequate power.

The null hypothesis (H₀) is the default statistical proposition that there is no effect, no difference, or no relationship between variables. It serves as the baseline assumption that an experiment aims to test against.

• The p-value is calculated under the assumption that the null hypothesis is true.
• A low p-value indicates that the observed data is unlikely under H₀, providing evidence to reject the null hypothesis.
• Example: In an A/B test, H₀ states that the new model variant (B) has the same conversion rate as the current model (A).

Statistical significance is a determination that an observed effect in sample data is unlikely to have occurred by random chance alone. It is formally declared when a p-value falls below a pre-defined significance level (alpha, α).

• Common alpha levels are 0.05 (5%) or 0.01 (1%).
• A result with p < 0.05 is termed "statistically significant."
• Crucial Note: Statistical significance does not imply practical importance or the size of the effect. A tiny, irrelevant effect can be statistically significant with a large enough sample.

A confidence interval (CI) provides a range of plausible values for an unknown population parameter (like the true difference between two model variants), estimated from sample data. A 95% CI means that if the experiment were repeated many times, 95% of such intervals would contain the true parameter value.

• It is directly related to significance testing: A 95% CI that does not include zero (e.g., [0.5%, 3.2%]) corresponds to a p-value < 0.05 for a test of no difference.
• CIs convey more information than a binary p-value, as they indicate both the precision (width of the interval) and the magnitude of the estimated effect.

These are the two fundamental errors in hypothesis testing, framed in relation to the null hypothesis.

• Type I Error (False Positive): Incorrectly rejecting a true null hypothesis. The probability of committing a Type I error is equal to the significance level α. A p-value threshold of 0.05 accepts a 5% chance of a false positive.
• Type II Error (False Negative): Failing to reject a false null hypothesis. The probability of a Type II error is denoted by β.
• Statistical Power is (1 - β), the probability of correctly rejecting a false null hypothesis. High power reduces the risk of missing a real effect.

Statistical power is the probability that a test will correctly reject a false null hypothesis. It is a test's sensitivity to detect a true effect of a specified size.

• Power is influenced by three key factors:
  • Sample Size: Larger samples increase power.
  • Effect Size: Larger true effects are easier to detect (higher power).
  • Significance Level (α): A larger α (e.g., 0.10 vs. 0.05) increases power but also increases the false positive rate.
• Underpowered experiments (low power) are a major pitfall, as they are likely to produce non-significant p-values even when a meaningful effect exists.

Bayesian inference is an alternative statistical paradigm to the frequentist methods that produce p-values. Instead of assessing the probability of data given a hypothesis (the p-value), it calculates the probability of a hypothesis given the data.

• It combines prior beliefs about parameters with observed data to form a posterior distribution.
• Results are expressed as credible intervals (the Bayesian analog to confidence intervals) and direct probabilities for hypotheses (e.g., "There is a 92% probability that Variant B is better").
• This framework avoids some common misinterpretations of p-values and naturally incorporates prior knowledge, but requires specifying a prior distribution.