Calibration Set

A heuristic function that scores how unusual a prediction is relative to the calibration data. This is the engine of conformal prediction.

• Input: A data point and a candidate label.
• Output: A scalar score where higher values indicate greater nonconformity.
• Common examples: Absolute residual for regression, 1 minus softmax probability for classification.

The calibration set is used exclusively to compute the empirical distribution of these scores, which directly determines the threshold for inclusion in the final prediction set.

The most computationally efficient variant that explicitly requires a held-out calibration set distinct from the training data.

• Proper Training Set: Used to fit the underlying model exactly once.
• Calibration Set: Used to compute nonconformity scores and the empirical quantile.
• Key Advantage: Avoids the prohibitive cost of retraining the model multiple times, making it suitable for deep learning.

This method partitions the available labeled data, reserving a portion strictly for calibration to ensure the prediction sets have valid marginal coverage.

The fundamental statistical assumption that makes the calibration set valid. It requires that the joint distribution of the calibration and test data is invariant to permutation.

• Weaker than IID: It does not require independence, only symmetry.
• Practical meaning: The order of data points doesn't matter; no single point is special.
• Violation example: Time series data with strong temporal trends breaks exchangeability.

If this assumption holds, the rank of a test point's nonconformity score among the calibration scores is uniformly distributed, guaranteeing the coverage probability.

The core statistical promise delivered by the calibration set methodology. It guarantees that P(Y_test ∈ Prediction Set) ≥ 1 - α.

• Marginal: The probability is averaged over the randomness in both the calibration and test data draws.
• Not conditional: It does not guarantee coverage for a specific input feature value, only on average across the population.
• Distribution-free: This guarantee holds for any underlying model or data distribution, provided exchangeability holds.

The size of the calibration set directly impacts the precision of this guarantee; larger sets yield tighter bounds.

A technique that wraps a standard quantile regression model with a conformal calibration step to correct its predicted intervals.

• Initial intervals: The base model predicts lower and upper quantiles (e.g., 5th and 95th percentiles).
• Calibration step: The calibration set is used to compute a correction factor for these intervals based on the nonconformity of the quantile predictions.
• Result: Produces finite-sample valid prediction intervals without assuming a specific error distribution, fixing any under-coverage of the raw quantile regressor.

The application of conformal p-values to the unsupervised problem of outlier detection, using a calibration set of normal data.

• Calibration Set: Composed exclusively of known normal, non-anomalous instances.
• Nonconformity Measure: Often based on distance to nearest neighbors or reconstruction error from an autoencoder.
• Decision Rule: A test point is flagged as anomalous if its conformal p-value falls below a predefined significance level ε.

This provides rigorous control over the false positive rate, guaranteeing that the probability of falsely flagging a normal point is at most ε.