Conformal Prediction

The core property of conformal prediction is the finite-sample, distribution-free guarantee of marginal coverage. For a user-specified error rate α (e.g., 0.1), the method ensures that the true label is included in the prediction set with probability at least 1-α.

• No asymptotic assumptions: The guarantee holds for any sample size, not just in the limit
• Distribution-free: No assumptions about the underlying data distribution are required
• Model-agnostic: Works with any pre-trained classifier, from neural networks to random forests
• The guarantee is marginal over both training and test data, meaning coverage is averaged over future test points

The engine of conformal prediction is the nonconformity score, a scalar measure of how unusual a potential label is for a given input relative to a calibration dataset. Common choices include:

• 1 - softmax probability for classification: higher scores indicate less conformity
• Absolute residual for regression: the difference between predicted and true values
• Adaptive prediction sets (APS): accumulates sorted class probabilities until a threshold is met
• The score function is the only design choice; the coverage guarantee holds regardless of its quality, though better scores yield smaller, more informative prediction sets

Split conformal prediction is the most practical variant, dividing available data into a proper training set and a held-out calibration set to avoid retraining the model.

• Training set: Used to fit the underlying model exactly once
• Calibration set: Used to compute nonconformity scores and determine the empirical quantile threshold
• Computationally efficient: No model retraining required, unlike full or jackknife+ methods
• The calibration set must be exchangeable with test data for the coverage guarantee to hold
• Typical calibration splits range from 20-30% of available labeled data

Instead of outputting a single class label, conformal prediction produces a prediction set—a subset of possible labels guaranteed to contain the true label with high probability.

• Binary classification: Sets may be {0}, {1}, or {0,1} (indicating uncertainty)
• Multi-class: Sets adapt in size based on difficulty; ambiguous inputs yield larger sets
• Regression: Produces prediction intervals [L(X), U(X)] with guaranteed coverage
• Set size serves as an interpretable uncertainty metric: larger sets indicate higher model uncertainty
• Enables safe abstention: if the set contains multiple classes, the system can defer to a human expert

While marginal coverage is guaranteed, achieving conditional coverage—coverage within specific subgroups or input regions—is a more challenging goal that standard conformal prediction does not automatically satisfy.

• Marginal coverage: P(Y_test ∈ C(X_test)) ≥ 1-α, averaged over all test points
• Conditional coverage: P(Y_test ∈ C(X_test) | X_test = x) ≥ 1-α for every individual x
• Standard split conformal may undercover certain subpopulations while overcovering others
• Mondrian conformal prediction extends the framework to provide coverage guarantees within pre-defined categories or strata
• Class-conditional conformal ensures coverage per class, critical for imbalanced classification tasks

Conformal prediction offers a principled approach to adversarial detection and robust classification by quantifying uncertainty in a way that is sensitive to distributional shifts caused by perturbations.

• Detection via set size: Adversarial inputs often produce abnormally large or empty prediction sets, serving as a detection signal
• Certified robustness with smoothing: Combining randomized smoothing with conformal prediction yields prediction sets with provable coverage under Lp-norm bounded perturbations
• Out-of-distribution rejection: Nonconformity scores naturally flag inputs far from the training distribution
• The framework complements adversarial training by providing statistical guarantees alongside empirical robustness improvements

The core output of a conformal predictor. Instead of a single class label, the model returns a set of plausible labels guaranteed to contain the true label with a user-specified probability (e.g., 95%).

• Singleton set: High confidence in a single class
• Empty set: Input is anomalous or out-of-distribution
• Multi-class set: Model is uncertain between several options

The set size directly reflects epistemic uncertainty, making it ideal for spectrum monitoring where ambiguous signals require human analyst review.

A real-valued function that scores how atypical a new example is relative to a calibration dataset. This score drives the entire conformal framework.

Common measures in signal classification:

• 1 - softmax probability of the true class
• Distance to nearest centroid in feature space
• Residual magnitude from a regression model

The choice of nonconformity measure directly impacts prediction set efficiency—a good measure yields smaller, more informative sets.

A held-out dataset, never seen during training, used exclusively to compute nonconformity scores and establish the empirical quantile threshold.

Key properties:

• Must be exchangeable with future test points
• Typically 10-20% of available labeled data
• Determines the finite-sample validity of coverage guarantees

In adversarial RF environments, maintaining calibration set integrity is critical—a poisoned calibration set invalidates all coverage guarantees.

The formal probabilistic statement that, over repeated trials, the prediction set will contain the true label at least 1 - α of the time, where α is the user-specified error rate.

Critical nuance: This is a marginal guarantee—coverage holds on average across all test points, not conditionally for each individual input. Conditional coverage remains an active research challenge.

For modulation classification, this means a conformal predictor set at α=0.05 will miss the true modulation type at most 5% of the time, regardless of the underlying model architecture.

A computationally efficient variant that splits the training process into two phases, avoiding the need to retrain the underlying model for every new test point.

Workflow:

• Train the base model once on the proper training set
• Compute nonconformity scores on the calibration set
• For each test point, compute its score and compare to the calibration quantile

This makes conformal prediction practical for real-time spectrum classification where latency budgets are measured in microseconds.

The fundamental statistical requirement that the joint distribution of calibration and test data is invariant under permutation. In simpler terms: the order of data points doesn't matter.

Violations in RF domains:

• Temporal distribution shift (changing channel conditions)
• Adversarial data injection
• Non-stationary interference patterns

When exchangeability breaks, coverage guarantees degrade. Adaptive conformal inference techniques use online error monitoring to adjust thresholds dynamically and recover validity.