Conformal OOD Detection

The core mechanism computes a conformal p-value for each test input by comparing its nonconformity score against the empirical distribution of scores from a held-out calibration set of in-distribution data. A small p-value indicates the sample is significantly nonconforming relative to the training distribution, flagging it as OOD. Unlike raw anomaly scores, p-values are calibrated probabilities with a uniform distribution under the null hypothesis of in-distribution data, enabling principled threshold selection.

The defining advantage of conformal OOD detection is the finite-sample guarantee on the Type I error rate. By setting a significance level ε (e.g., 0.05), the framework guarantees that the probability of falsely flagging an in-distribution sample as OOD is exactly ε, regardless of the underlying model or data distribution. This holds under the exchangeability assumption between calibration and test in-distribution data. Key properties:

• Model-agnostic: Works with any pre-trained classifier or density estimator
• Distribution-free: No parametric assumptions about the data
• Finite-sample valid: Guarantee holds for any calibration set size

The effectiveness of OOD detection hinges on the choice of nonconformity measure—a function scoring how atypical an input is. Common designs include:

• Softmax probability: 1 minus the maximum predicted class probability from a classifier
• Energy-based scores: The negative free energy from an energy-based model
• Mahalanobis distance: Distance to class-conditional Gaussians in feature space
• Deep ensemble variance: Disagreement among ensemble members
• Learned nonconformity: Training a dedicated scoring network on calibration residuals The measure must capture semantic novelty, not just pixel-level variation.

The standard implementation uses split conformal prediction to avoid retraining. The procedure:

1. Training phase: Fit the base model on a proper training set
2. Calibration phase: Compute nonconformity scores on a disjoint calibration set of in-distribution data
3. Threshold computation: Calculate the (1-ε) empirical quantile of calibration scores
4. Test phase: For each new input, compute its nonconformity score and derive a p-value by comparing it to the calibration distribution This decoupling ensures computational efficiency at test time with no model retraining required.

Conformal OOD detection naturally extends to open-set recognition where test inputs may belong to unknown classes. The framework can be applied per-class using Mondrian conformal prediction, computing class-conditional p-values that control the false positive rate within each known category. A test sample is classified as OOD if it is rejected by all known-class detectors simultaneously. This provides label-conditional guarantees and avoids the asymmetry of one-vs-all OOD scoring.

Standard conformal OOD detection assumes exchangeability between calibration and test in-distribution data. When this assumption is violated by covariate shift or gradual data drift, adaptive conformal inference techniques can be applied. These methods dynamically update the detection threshold online using a sliding window of recent observations or by applying importance weights to calibration samples. This maintains approximate false positive rate control even as the in-distribution data evolves over time.

The direct application of conformal p-values to the unsupervised problem of outlier detection. It operates by defining a nonconformity measure (e.g., reconstruction error from an autoencoder or distance in a latent space) on a calibration set of normal data. A new test point is flagged as out-of-distribution if its nonconformity score is significantly higher than the calibration scores, with the false positive rate explicitly controlled by the user-specified significance level.

The core heuristic function A(B, z) that quantifies how strange a data point z looks relative to a bag of training examples B. In OOD detection, this is often the negative log-likelihood from a density model, the Mahalanobis distance to a class centroid, or the energy score from a discriminative classifier. The quality of the OOD detector is entirely dependent on choosing a measure that assigns high scores to semantic novelties and low scores to in-distribution data.

A specific architecture where an autoencoder is trained solely on in-distribution data to minimize reconstruction error. The reconstruction error on a held-out calibration set forms the empirical distribution of nonconformity scores. A conformal threshold is then computed to guarantee that the probability of falsely flagging a normal input as OOD is bounded by α. This provides a rigorous upgrade over heuristic thresholding of reconstruction errors.

The computationally efficient framework that underpins most conformal OOD detectors. The data is split into three disjoint sets:

• Proper training set: Used to fit the model (e.g., a classifier or density estimator).
• Calibration set: Used exclusively to compute nonconformity scores and the empirical quantile threshold.
• Test set: New data evaluated against the fixed threshold. This avoids the prohibitive cost of retraining the model for every new test point.

The statistical output of a conformal OOD test. For a test object z_{n+1}, the conformal p-value is the fraction of calibration nonconformity scores that are greater than or equal to the score of z_{n+1}. A small p-value (e.g., < 0.05) indicates the point is highly nonconforming and thus likely OOD. Crucially, if the test point is truly in-distribution, the p-value is uniformly distributed on [0,1], providing a rigorous false detection rate control.

A technique applied to OOD detection to achieve class-conditional validity. Instead of computing a single global threshold, Mondrian conformal prediction partitions the calibration data by class label and computes a separate nonconformity threshold for each class. This prevents a model from masking OOD inputs that fall near a dense, high-variance class by ensuring rigorous detection guarantees for every in-distribution category independently.