Conformal Prediction

Conformal prediction operates as a post-hoc calibration layer that wraps around any pre-trained model—whether a deep neural network, gradient-boosted tree, or linear regression—without requiring access to model internals or retraining.

• Works with black-box models: only requires model outputs, not gradients or architecture
• Compatible with classification and regression tasks alike
• Preserves the underlying model's accuracy while adding rigorous uncertainty bounds
• Example: Wrap a Temporal Fusion Transformer demand forecast with conformal intervals without modifying the attention layers

Unlike asymptotic confidence intervals that hold only as sample size approaches infinity, conformal prediction provides exact marginal coverage with a finite calibration set. If you request 90% coverage, at least 90% of true values fall within the prediction intervals.

• Guarantee holds regardless of model complexity or underlying distribution
• Coverage probability is distribution-free—no Gaussian assumptions required
• The only assumption: exchangeability of calibration and test data points
• Critical for supply chain applications where demand distributions are heavy-tailed or multi-modal

The core mechanism of conformal prediction is the nonconformity score—a heuristic that measures how unusual a potential label is relative to the calibration data. Common choices include:

• Absolute residual: |y - ŷ| for regression, producing fixed-width intervals
• Normalized residual: |y - ŷ| / σ(x) for adaptive intervals that widen in high-uncertainty regions
• Softmax probability: 1 - p(y|x) for classification, yielding prediction sets instead of single labels
• The choice of nonconformity score directly shapes interval adaptivity and efficiency

Two primary algorithmic variants balance computational cost against statistical efficiency:

• Split (inductive) conformal: Partitions data into a training set and a calibration set. The model trains once; calibration computes quantiles of nonconformity scores. Computationally cheap but sacrifices some data for calibration.
• Full (transductive) conformal: Retrains the model for every candidate label, exhaustively evaluating all possibilities. Statistically optimal but computationally prohibitive for deep learning.
• Cross-conformal and jackknife+ methods offer intermediate trade-offs, using leave-one-out or k-fold strategies to improve data efficiency

Standard conformal regression produces intervals of constant width across the input space. Adaptive conformal methods use normalized nonconformity scores to produce intervals that automatically widen in regions of high epistemic uncertainty and narrow where the model is confident.

• Requires an auxiliary model to estimate conditional variance σ(x)
• Intervals reflect heteroscedastic noise—critical for intermittent demand patterns
• Example: A demand forecast for a stable product gets tight bounds; a volatile promotional item receives appropriately wide intervals
• Enables differentiated safety stock policies per SKU based on forecast confidence

Combines quantile regression with conformal calibration to produce distribution-free prediction intervals that inherit the adaptivity of quantile estimates while correcting any coverage deficiencies.

• Train lower (α/2) and upper (1-α/2) quantile models on training data
• Calibrate the raw quantile intervals on a held-out set using conformal adjustment
• The resulting intervals maintain the shape flexibility of quantile regression with guaranteed coverage
• Outperforms both raw quantile regression and standard conformal methods on heteroscedastic time series

A prediction interval is a range of values within which a future observation is expected to fall with a specified confidence level (e.g., 90%). Unlike a confidence interval, which estimates a population parameter, a prediction interval quantifies the uncertainty of a single future realization. In conformal prediction, these intervals are constructed without assuming a specific data distribution.

• Key distinction: Predicts a single future value, not a population mean
• Coverage guarantee: A 90% prediction interval should contain the true value 90% of the time
• Application: Setting dynamic safety stock levels that achieve a target service level

Conformal prediction provides coverage guarantees that account for both types of uncertainty. Aleatoric uncertainty is the irreducible noise inherent in the data-generating process—the natural randomness of customer demand. Epistemic uncertainty is the reducible uncertainty from limited data or model misspecification, which shrinks as more training samples are collected.

• Aleatoric: Constant-width prediction intervals even with infinite data
• Epistemic: Wider intervals in sparse data regions, narrower in dense regions
• Adaptive conformal prediction: Produces intervals that automatically widen in high-uncertainty regions

Backtesting is the process of evaluating a forecasting model by applying it to historical data using a rolling or expanding window. For conformal prediction, backtesting validates that the empirical coverage matches the nominal coverage over time. A well-calibrated conformal predictor should achieve a 90% coverage rate when backtested across multiple time periods.

• Rolling window: Train on periods 1–T, test on T+1; then slide forward
• Coverage tracking: Monitor the running average of interval hits vs. misses
• Drift detection: A sudden drop in empirical coverage signals concept drift

Safety stock optimization determines the optimal buffer inventory to achieve a target service level under demand and supply variability. Conformal prediction directly informs this process by providing valid prediction intervals for lead time demand. A 95% conformal prediction interval translates to a 95% service level when used to set reorder points.

• Newsvendor model: Balances overstocking cost against understocking cost
• Dynamic safety stock: Intervals adapt to changing demand volatility in real time
• Multi-echelon: Conformal intervals can be propagated through supply chain tiers