Distribution Calibration

Unlike quantile calibration, which only verifies specific percentiles, distribution calibration demands that the predicted probability integral transform (PIT) values are uniformly distributed. A model is distributionally calibrated if, for all confidence levels α, the fraction of true outcomes falling below the predicted α-quantile equals α. This ensures the entire predictive CDF is statistically indistinguishable from the true data-generating process.

The PIT is the foundational diagnostic tool for distribution calibration. It transforms observed outcomes using the predicted CDF: if the predicted distribution is correct, the transformed values follow a Uniform(0,1) distribution.

• Uniform PIT histogram: Indicates perfect calibration
• U-shaped PIT histogram: Indicates underdispersed predictions (too narrow)
• Hump-shaped PIT histogram: Indicates overdispersed predictions (too wide)

Deviations from uniformity immediately reveal the nature of miscalibration.

Distribution calibration is evaluated using strictly proper scoring rules that are minimized only when the predicted distribution matches the true distribution:

• Continuous Ranked Probability Score (CRPS): Generalizes the Brier Score to continuous domains by measuring the integrated squared difference between the predicted CDF and the empirical step function at the observation
• Log Score: The negative log of the predictive density evaluated at the observed outcome, heavily penalizing predictions that assign near-zero probability to realized events

Both metrics assess calibration and sharpness simultaneously.

Distribution calibration is the strongest form of calibration in the regression hierarchy:

• Marginal calibration: Only ensures the average prediction matches the average outcome (weakest)
• Quantile calibration: Ensures each predicted quantile matches empirical frequency (stronger)
• Distribution calibration: Ensures the entire predictive distribution is correct (strongest)

A distributionally calibrated model is automatically quantile-calibrated for all quantiles, but the converse is not true. A model can pass quantile checks at the 5th, 50th, and 95th percentiles while still having a misspecified distribution shape between those points.

Several techniques can transform uncalibrated predictive distributions into distributionally calibrated ones:

• Isotonic distributional regression: Learns a monotonic transformation of the CDF that minimizes calibration error on a held-out calibration set
• Conformalized distributional prediction: Wraps any distribution predictor with conformal inference to produce calibrated prediction sets with finite-sample guarantees
• Beta calibration for probabilities: When outputs are bounded in [0,1], beta calibration learns a family of recalibration maps parameterized by a beta distribution

All methods require a dedicated calibration dataset distinct from training and test data to avoid overfitting.

Distribution calibration is critical in domains where decisions depend on the full probability distribution rather than point estimates:

• Financial risk management: Value-at-Risk (VaR) and Expected Shortfall models must have calibrated tail distributions to correctly estimate extreme loss probabilities
• Weather forecasting: Ensemble prediction systems must produce calibrated forecast distributions for precipitation, temperature, and wind speed
• Supply chain demand planning: Safety stock calculations require calibrated demand distributions to achieve target service levels without excess inventory

In each case, miscalibration in the tails can lead to catastrophic underestimation of risk.

The primary empirical metric for diagnosing miscalibration. ECE partitions predictions into M equally-spaced bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin.

• Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|
• Weakness: Sensitive to binning strategy; does not detect distribution-level miscalibration in regression.
• Target: An ECE of 0 indicates perfect calibration.

A powerful non-parametric calibration method that learns a piecewise constant, monotonically increasing function to map uncalibrated scores to empirical probabilities.

• Advantage: Makes no assumptions about the functional form of the miscalibration, unlike Platt scaling.
• Mechanism: Solves a constrained optimization problem to minimize a squared error loss subject to a monotonicity constraint.
• Risk: Prone to overfitting on small datasets due to its high flexibility; requires a dedicated calibration set.

Loss functions that are uniquely minimized when the predicted probability distribution exactly matches the true data-generating distribution. They encourage honest forecasting.

• Brier Score: Mean squared error between predicted probability and binary outcome. Sensitive to both calibration and refinement.
• Negative Log-Likelihood (NLL): Penalizes confident misclassifications heavily; a strictly proper scoring rule.
• Key Property: A forecaster has no incentive to deviate from their true belief when evaluated by a proper scoring rule.

A distribution-free, model-agnostic framework that wraps any predictor to produce prediction sets with a finite-sample, marginal coverage guarantee.

• Guarantee: For a user-specified error rate α, the true label is included in the prediction set with probability ≥ 1 - α.
• Mechanism: Uses a calibration set to compute nonconformity scores, which determine the threshold for inclusion.
• Application: Provides rigorous statistical guarantees for regression intervals and classification sets without assuming a specific data distribution.

The systematic process of characterizing and separating total predictive uncertainty into its constituent parts to assess prediction reliability.

• Aleatoric Uncertainty: Irreducible noise inherent in the data (e.g., sensor noise). Cannot be reduced with more data.
• Epistemic Uncertainty: Reducible model uncertainty due to lack of knowledge. High in out-of-distribution regions; decreases with more representative training data.
• Methods: Deep Ensembles, Monte Carlo Dropout, and Evidential Deep Learning are key techniques for UQ.