Prediction Interval Coverage Probability (PICP)

PICP is calculated as the ratio of test points captured within the prediction interval to the total number of test points.

• Formula: PICP = (1 / N) * Σ c_i where c_i = 1 if the true target y_i is within the lower and upper bounds [L_i, U_i], and 0 otherwise.
• Target: For a 95% prediction interval, the ideal PICP should be statistically indistinguishable from 0.95.
• Interpretation: A PICP significantly lower than the nominal confidence level indicates the intervals are too narrow and the model is overconfident.

PICP must always be evaluated alongside the Mean Prediction Interval Width (MPIW) to prevent trivial solutions.

• The Trade-off: A model can achieve a perfect PICP of 1.0 by outputting infinitely wide intervals, which is useless for decision-making.
• Sharpness vs. Coverage: The goal is to maximize PICP (coverage) while minimizing MPIW (sharpness).
• Combined Metric: The Coverage Width-Based Criterion (CWC) penalizes intervals that fail to meet the target PICP while rewarding narrow intervals that do.

PICP measures marginal coverage—the average coverage over the entire test distribution. This is distinct from conditional coverage.

• Marginal Coverage: Guarantees coverage on average across all inputs. A 95% PICP means 95% of all points are captured globally.
• Conditional Coverage: Requires that the interval covers the true value with the nominal probability for each specific input x. This is a much stronger, often unattainable guarantee.
• Practical Implication: A model with perfect marginal PICP can still have systematic coverage failures in specific sub-regions of the feature space.

PICP serves as a critical diagnostic tool for validating Uncertainty Quantification (UQ) methods in regression tasks.

• Calibration Check: It directly validates if the model's predicted uncertainty is calibrated. A 90% interval should contain the truth 90% of the time.
• Method Comparison: It provides a single, interpretable number to compare different UQ techniques like Deep Ensembles, Monte Carlo Dropout, or Quantile Regression.
• Failure Detection: A sudden drop in PICP on a new data batch can signal covariate shift or data quality issues in a production monitoring pipeline.

Relying solely on PICP can be misleading without understanding its statistical limitations.

• Finite Sample Error: PICP is an empirical estimate. With a small test set, the observed PICP can deviate significantly from the true coverage probability due to sampling noise.
• Sharpness Blindness: PICP ignores interval width entirely. A model with a high PICP but excessively wide intervals is poorly calibrated for practical use.
• Non-Adaptivity: It does not reveal if intervals are inappropriately wide in low-noise regions and too narrow in high-noise regions, masking heteroscedastic miscalibration.

PICP is the empirical measure that validates the finite-sample, distribution-free coverage guarantee provided by Conformal Prediction.

• Guarantee: Split conformal prediction mathematically guarantees that PICP ≥ 1 - α on exchangeable test data, where α is the user-specified error rate.
• Validation: PICP is the metric used to empirically confirm this guarantee holds on a held-out calibration set.
• Exact Coverage: Unlike asymptotic methods, conformal prediction's coverage guarantee is exact and non-asymptotic, making PICP a verification of a proven statistical property rather than just an aspiration.