Out-of-Distribution Detection

OOD detection relies on quantifying the epistemic uncertainty of a model's prediction. Unlike softmax probabilities, which can be high for OOD inputs, true uncertainty metrics identify when a model is operating outside its knowledge. Bayesian neural networks and deep ensembles model weight distributions to produce predictive variance. High variance across ensemble members signals an unfamiliar input. Monte Carlo Dropout approximates Bayesian inference during inference by sampling multiple forward passes, measuring disagreement as a proxy for distributional shift.

These techniques operate on the principle that in-distribution (ID) samples cluster tightly in the model's feature embedding space, while OOD samples map to distant, low-density regions. Key approaches include:

• Mahalanobis Distance: Computes the distance of a test sample's feature vector to the nearest class-conditional Gaussian distribution, using a tied covariance matrix estimated from training data.
• Deep Nearest Neighbors: Compares the test embedding against a stored gallery of ID embeddings using k-NN; a low conformity score triggers rejection.
• Gaussian Mixture Models (GMMs): Fit a probabilistic density model to the ID embeddings and flag samples with low log-likelihood.

These methods derive OOD scores directly from a model's logit outputs without modifying the architecture. The Energy Score computes the negative LogSumExp of the logits, providing a scalar that is theoretically aligned with the input's probability density. ID samples yield lower (more negative) energy. Maximum Softmax Probability (MSP) uses the highest softmax value as a confidence proxy, though it is known to be poorly calibrated for OOD detection. ODIN (Out-of-DIstribution detector for Neural networks) enhances MSP by applying temperature scaling and small input perturbations to widen the gap between ID and OOD scores.

Proactive techniques modify the training objective to enforce a compact decision boundary for ID classes and a repulsive margin for everything else. Outlier Exposure (OE) introduces a diverse auxiliary dataset of OOD examples during training, penalizing the model for making confident predictions on them. Contrastive Learning pulls augmented views of the same I/Q sample together while pushing all other samples apart, creating a highly structured embedding space. CAC (Classifier-Agnostic Contrastive) loss explicitly optimizes for a uniform distribution of OOD samples on the embedding hypersphere.

OpenMax replaces the standard softmax layer with a mechanism that estimates the probability of an input belonging to an unknown class. It fits a Weibull distribution to the distances between correctly classified training samples and their class mean activation vectors. At inference, the model recalibrates the logits by weighting them against the probability of extreme distance values, redistributing mass to a new 'unknown' pseudo-class. This explicitly models the boundary between known modulation schemes and the open world of novel waveforms.

Standard classification accuracy is insufficient for evaluating OOD detectors. Key metrics include:

• AUROC (Area Under the Receiver Operating Characteristic curve): Measures the trade-off between true positive rate (correctly identifying OOD) and false positive rate (flagging ID as OOD).
• FPR at 95% TPR (FPR95): The false positive rate when the true positive rate is fixed at 95%. Lower is better.
• AUPR (Area Under the Precision-Recall curve): More informative than AUROC when there is a class imbalance between ID and OOD samples.
• Detection Error: The minimum misclassification probability achievable by tuning the threshold.

A classification paradigm where the model must not only classify known modulation schemes but also detect and reject unknown modulation types not seen during training. Unlike closed-set AMC, which forces every input into a known class, open-set recognition establishes a decision boundary around known classes.

• Rejects novel waveforms rather than misclassifying them
• Critical for electronic warfare environments with unknown emitters
• Often implemented using extreme value theory to model class boundaries

A probabilistic output, often derived from a softmax layer or log-likelihood ratio, that quantifies the classifier's certainty in its prediction. Low confidence scores serve as a primary indicator of OOD samples.

• Enables downstream systems to make risk-aware decisions
• Can be calibrated using temperature scaling for reliability
• Threshold-based rejection: scores below a tuned threshold flag unknowns

A parametric OOD detection method that computes the distance between a test sample's feature representation and the class-conditional Gaussian distribution of each known class. Unlike softmax confidence, this metric operates in the penultimate layer's feature space.

• Accounts for feature covariance, not just Euclidean distance
• Requires fitting a mean vector and covariance matrix per class
• Computationally lightweight for real-time deployment

A scoring function that uses the Helmholtz free energy formulation of a neural network to separate in-distribution from OOD samples. Energy scores consistently outperform softmax confidence for OOD detection across architectures.

• Derived from the log of the denominator in the softmax function
• Lower energy indicates higher likelihood of being in-distribution
• Does not require OOD data during training

A self-supervised training method that learns robust signal representations by pulling augmented views of the same I/Q sample together and pushing views from different samples apart in the embedding space. These compact, well-separated clusters naturally improve OOD detection.

• Creates tighter intra-class clusters for known modulations
• OOD samples fall into low-density regions of the embedding
• Augmentations include noise addition, phase rotation, and fading

The resilience of a trained AMC model against intentionally crafted, minimal perturbations to the input signal designed to cause misclassification. OOD detection and adversarial robustness are closely linked: adversarial examples often appear as OOD samples in feature space.

• Gradient-based attacks can push in-distribution signals across decision boundaries
• Adversarial training can improve both robustness and OOD detection
• Critical security concern in contested electromagnetic environments