Conformal Prediction

The foundational property of conformal prediction: for any user-specified significance level α, the prediction set will contain the true label with probability at least 1-α. This is a finite-sample guarantee—it holds for any dataset size, not just asymptotically. The guarantee is marginal, meaning it holds on average over calibration and test data, not conditionally for every input. For open set emitter recognition, this provides a statistically valid mechanism to bound the false acceptance rate of unknown classes.

Conformal prediction requires no assumptions about the data distribution. Unlike Gaussian processes or Bayesian neural networks that assume normality, conformal methods work with any underlying distribution—heavy-tailed, multimodal, or adversarial. This is critical for RF fingerprinting where signal impairments create complex, non-Gaussian feature distributions. The only requirement is exchangeability: the calibration and test data must be drawn from the same distribution, a condition satisfied by standard i.i.d. splits.

The framework is agnostic to the underlying model and scoring function. Any nonconformity measure can be used:

• Adaptive Prediction Sets (APS): Uses cumulative softmax probabilities, producing smaller sets for easy examples
• Regularized Adaptive Prediction Sets (RAPS): Adds penalty for set size, optimizing efficiency
• Distance-based scores: Uses Mahalanobis distance or embedding proximity for open set rejection This flexibility allows practitioners to optimize for set size efficiency while maintaining the coverage guarantee.

Conformal prediction operates through a strict split-conformal procedure:

1. Training: Fit the base model on training data
2. Calibration: Compute nonconformity scores on a held-out calibration set to determine the empirical quantile threshold
3. Prediction: For each test point, include all labels with scores below the calibrated threshold The calibration set must remain untouched during training to preserve exchangeability. This protocol is computationally lightweight, requiring only a single forward pass over calibration data.

In open set emitter recognition, conformal prediction naturally handles unknown classes: if the prediction set is empty or excludes all known classes, the input is flagged as unknown. This provides a principled alternative to heuristic thresholding. By controlling α, operators can directly trade off between false unknown rejection and false known acceptance. For spectrum surveillance, this enables statistically validated emitter identification with guaranteed false alarm rates.

While marginal coverage is guaranteed, conditional coverage—validity for each specific input—is impossible to achieve distribution-free. This limitation means conformal sets may be systematically too large for easy examples and too small for hard ones. Advanced variants address this:

• Mondrian conformal prediction: Provides coverage guarantees within predefined strata or classes
• Conformalized quantile regression: Achieves approximate conditional coverage by wrapping quantile regression models For RF applications, stratifying by signal-to-noise ratio or modulation type can improve practical reliability.

The process of aligning a model's predicted probability of correctness with its actual empirical accuracy. A perfectly calibrated model will have a confidence of 0.9 be correct 90% of the time. Conformal prediction provides a distribution-free alternative to calibration, guaranteeing marginal coverage without requiring the model's probabilities to be meaningful.

• Temperature Scaling: A post-hoc method that softens SoftMax outputs
• Expected Calibration Error (ECE): Measures the gap between confidence and accuracy
• Unlike conformal prediction, calibration does not guarantee set coverage

The risk of labeling an unknown sample as a known class, quantified by the volume of space far from training data that is nonetheless classified as known. Conformal prediction directly mitigates this risk by producing empty or large prediction sets for inputs far from the training distribution.

• Defined by the ratio of open space to total feature space
• Traditional SoftMax classifiers have unbounded open space risk
• Conformal sets naturally expand in regions of high epistemic uncertainty

The reducible model uncertainty arising from a lack of knowledge or data. This is high for inputs far from the training distribution and is the primary signal used to reject unknown classes. Conformal prediction captures epistemic uncertainty through the size of the prediction set.

• Large prediction sets indicate high epistemic uncertainty
• Empty prediction sets signal out-of-distribution inputs
• Contrasts with aleatoric uncertainty, which is irreducible noise inherent in the data

A statistical discipline for modeling the probability of rare, extreme events. In open set recognition, EVT is used to calibrate rejection thresholds for unknown classes. Conformal prediction offers a complementary non-parametric approach that does not require fitting extreme value distributions.

• Weibull Calibration: Fits a Weibull distribution to class distances
• OpenMax: Uses EVT to replace SoftMax for open set rejection
• Conformal methods avoid distributional assumptions entirely

A low-dimensional vector representation of high-dimensional input data learned by a neural network, where semantic similarity is preserved as geometric proximity. Conformal prediction operates on these embeddings by computing nonconformity scores based on distances to class centroids or nearest neighbors.

• Prototypical Networks: Use class-mean embeddings for few-shot classification
• Angular Margin Loss (ArcFace, CosFace): Enforces discriminative constraints
• Nonconformity measures often use Mahalanobis distance or cosine similarity

The task of identifying input samples that differ significantly from the training data distribution, triggering a rejection mechanism. Conformal prediction provides a principled framework for OOD detection by producing empty or singleton prediction sets when the model is uncertain.

• Energy-Based Models: Assign high energy to OOD samples
• Deep SVDD: Maps normal data to a minimal-volume hypersphere
• Conformal anomaly detection uses p-values to flag distributional shifts