Conformal Prediction

Conformal prediction provides finite-sample, distribution-free validity. This means its coverage guarantees hold for any underlying data distribution and any finite sample size, without requiring assumptions like normality or large-sample asymptotics. The core guarantee is marginal coverage: for a user-chosen error rate α (e.g., 0.1), the method ensures that over many trials, the true label Y is contained in the prediction set C(X) at least 1-α of the time. This is a frequentist, non-asymptotic guarantee.

The framework operates as a wrapper around any underlying machine learning model (e.g., neural network, random forest, gradient boosting). It does not modify the model's internal architecture or training procedure. Instead, it uses the model's outputs (scores, distances, or probabilities) on a held-out calibration set to calculate a data-dependent threshold. This threshold is then applied to the model's outputs on new test points to form the prediction sets. This separation makes it highly versatile across different modeling paradigms.

Instead of outputting a single, potentially overconfident prediction, conformal prediction returns a set of plausible labels. For classification, this might be a subset of all possible classes (e.g., {cat, dog} instead of just cat). For regression, it outputs an interval. The size of the set conveys uncertainty: a large set indicates high ambiguity, while a singleton set indicates high confidence. This is more informative and safer for risk-sensitive applications than a point estimate with an uncalibrated confidence score.

This is the most computationally efficient and widely used variant. The procedure is:

• Split Data: Partition labeled data into a proper training set and a calibration set.
• Train Model: Fit any model on the training set.
• Compute Nonconformity Scores: Use the fitted model and a chosen nonconformity measure (e.g., 1 - predicted probability for the true label) to compute scores for each sample in the calibration set.
• Calculate Threshold: Determine the (1-α)-th quantile of these calibration scores.
• Form Prediction Sets: For a new test point, include all labels whose nonconformity score is less than or equal to this threshold.

The nonconformity measure is a function that quantifies how 'strange' or atypical a data point (x, y) is relative to the model's predictions. Common measures include:

• Classification: 1 - f(x)[y], where f(x)[y] is the model's predicted probability for the true label y.
• Regression: The absolute residual |y - f(x)|.
• Adaptive Measures: More sophisticated measures like Adaptive Prediction Sets (APS) or Regularized Adaptive Prediction Sets (RAPS) that can produce smaller, more efficient sets. The choice of measure directly influences the size and shape of the resulting prediction sets.

A critical nuance is that the standard guarantee is marginal coverage (average over all X). It does not guarantee conditional coverage for every X=x. In practice, coverage may be lower for some difficult subpopulations and higher for easier ones. Achieving valid conditional coverage is a major research challenge. Methods like conformalized quantile regression (CQR) for regression or approaches using weighted conformal prediction aim to provide better conditional properties. Practitioners must understand this limitation when deploying in settings requiring fairness or uniform reliability.

The overarching field of machine learning concerned with measuring and interpreting the different types of uncertainty inherent in a model's predictions. It provides the theoretical foundation for confidence scoring.

• Core Objective: To distinguish between aleatoric uncertainty (irreducible noise in the data) and epistemic uncertainty (reducible uncertainty from limited model knowledge).
• Methods: Includes Bayesian Neural Networks (BNNs), Monte Carlo Dropout, and Deep Ensembles.
• Relation to Conformal Prediction: While UQ provides probabilistic measures of uncertainty, conformal prediction uses these measures (or any model score) to construct statistically guaranteed prediction sets.

A measure of the discrepancy between a model's predicted confidence scores and its actual empirical accuracy. A well-calibrated model's confidence of 90% should correspond to being correct 90% of the time.

• Expected Calibration Error (ECE): The most common metric, calculated by binning predictions by confidence and averaging the absolute difference between average confidence and accuracy per bin.
• Diagnostic Tool: Visualized using a Reliability Diagram.
• Calibration Methods: Techniques like Platt Scaling and Temperature Scaling are used post-training to improve calibration.
• Critical Link: Conformal prediction does not require perfect calibration but uses calibration data to achieve its coverage guarantees, often correcting for miscalibration.

A paradigm where a model is allowed to abstain from making a prediction on inputs where its confidence is below a user-defined threshold. This trades coverage (the fraction of samples predicted on) for higher accuracy on the remaining set.

• Key Trade-off: Illustrated by a Risk-Coverage Curve, which plots error rate against the fraction of accepted samples.
• Relation to Conformal Prediction: Conformal prediction can be viewed as a generalization. Instead of a binary 'predict/abstain' decision, it outputs a prediction set that may contain multiple labels, with a guarantee that the set contains the true label. For classification, a singleton set is equivalent to a confident prediction, while a larger set indicates higher uncertainty/abstention.

In Bayesian statistics, a credible interval is a range of values within which an unobserved parameter (or a prediction) falls with a specified posterior probability. It is a probabilistic measure of uncertainty derived from a posterior distribution.

• Contrast with Conformal Intervals: A credible interval requires a correct Bayesian model and prior to have its stated probabilistic meaning asymptotically. A conformal prediction interval provides a distribution-free, finite-sample guarantee of coverage without requiring model correctness, making it more robust but often less efficient (wider) if the model is well-specified.

A specific application of the conformal prediction framework to regression tasks. It combines quantile regression models with conformal calibration to produce prediction intervals with guaranteed marginal coverage.

• Mechanism: A model is trained to predict two quantiles (e.g., the 5th and 95th). Conformal prediction then adjusts these quantile estimates on a calibration set to achieve the exact desired coverage level (e.g., 90%).
• Output: Produces an interval [low, high] for a regression target, guaranteeing that the true value lies within the interval with the user-specified probability.
• Use Case: The direct regression analogue to the classification sets produced by standard conformal prediction.

The task of identifying whether a given input sample is statistically different from the data distribution the model was trained on. This is a critical safety component, as models often make overconfident, incorrect predictions on OOD data.

• Connection to Uncertainty: OOD samples typically induce high epistemic uncertainty.
• Relation to Conformal Prediction: Conformal prediction's validity guarantee holds marginally over the calibration and test data, assuming they are exchangeable (i.e., from the same distribution). If a test sample is OOD, this assumption breaks, and coverage is not guaranteed. Thus, OOD detection is a crucial pre-filtering step for robust conformal prediction in open-world settings.