Out-of-Distribution Detection

OOD detection fundamentally operates by quantifying the divergence between the feature distribution of a test sample and the training data manifold. Mahalanobis distance provides a class-conditional covariance-aware metric, while Energy-Based Models assign a scalar energy score where in-distribution data occupies low-energy basins. The core mechanism involves fitting a probabilistic model—such as a Gaussian Mixture Model or Weibull distribution via Extreme Value Theory—to the logits or embeddings of known classes. Samples falling in the tail of this fitted distribution are flagged as OOD. This statistical rigor prevents the classifier from extrapolating high-confidence predictions into regions of the feature space where no training support exists.

The geometric arrangement of embeddings in the latent space is critical for separating known from unknown signals. Prototype Learning establishes a single representative vector per class, using distance to these prototypes as a novelty metric. Reciprocal Point Learning inverts this logic by representing classes with points distant from their manifold, flagging inputs that fail to be repelled by all reciprocal points. Feature Collapse is a catastrophic failure mode where all inputs—including unknowns—map to a restricted region, destroying discriminability. Effective OOD systems enforce a geometry where known classes form compact, separable clusters, leaving vast open space to be quantifiably rejected.

Distinguishing between epistemic uncertainty (model ignorance reducible with more data) and aleatoric uncertainty (inherent noise) is vital for OOD detection. Evidence Deep Learning places a Dirichlet distribution over class probabilities, directly outputting uncertainty mass that spikes for unknown inputs. Deep Ensembles measure predictive variance across multiple randomly initialized models, providing a robust uncertainty signal without architectural modification. Confidence Calibration via Temperature Scaling ensures that low SoftMax probabilities reliably correspond to incorrect or unknown inputs, preventing overconfident misclassifications in the open space.

Proactive OOD detection capability is instilled during training through specialized loss functions and data exposure. Outlier Exposure leverages an auxiliary dataset of diverse, non-target examples to force the network to learn a tighter decision boundary around known classes. Entropic Open-Set Loss compels the model to produce high-entropy, uniform output distributions for unknown samples, making them easily thresholdable. Objectosphere Loss manipulates feature magnitude by maximizing the L2 norm for known samples while minimizing it for unknowns, creating a clear magnitude gap for rejection. These methods fundamentally reshape the loss landscape to include a rejection objective.

Many OOD methods apply a scoring mechanism to a pre-trained model's outputs without retraining. The ODIN detector uses temperature scaling and small input perturbations to amplify the SoftMax score gap between in-distribution and OOD samples. The Maximum SoftMax Probability baseline simply thresholds the highest class probability, though it often fails on high-confidence unknowns. Energy-Based scoring computes the Helmholtz free energy from logits, providing a theoretically grounded score that outperforms SoftMax thresholding. These methods are computationally efficient but rely on the assumption that the base model's representations already encode useful separability.

Generative models offer an alternative paradigm by learning to reconstruct the training distribution and flagging deviations. Autoencoder Anomaly Detection trains a bottleneck network to minimize reconstruction error on known signals; OOD inputs, lacking the learned compressible structure, yield high reconstruction loss. Variational Autoencoders extend this by modeling the latent distribution, allowing likelihood-based detection. This approach is modality-agnostic and does not require class labels, making it suitable for unsupervised novelty detection. However, it assumes that reconstruction difficulty correlates with semantic novelty, which can fail if OOD samples share low-level statistical properties with the training set.

A classification paradigm where the model must accurately identify known classes while simultaneously detecting and rejecting samples from unknown or novel classes not seen during training. Unlike closed-set systems that fail silently, open set recognition explicitly models the possibility of encountering new modulation types in dynamic spectrum environments. The core challenge is balancing high accuracy on known signals with a low false acceptance rate for unknowns.

A statistical framework for modeling the tail behavior of distributions, used in open set recognition to fit a Weibull distribution to the distance of correct classifications from their class mean. EVT provides a principled way to estimate the probability that a sample belongs to an unknown class by analyzing the extreme values of the matching scores. This approach is foundational to the OpenMax layer, which replaces standard SoftMax with EVT-calibrated activation vectors.

A class of models that learn an energy function assigning low energy to in-distribution data and high energy to out-of-distribution data. Using the Helmholtz free energy as a discriminative score, these models provide a theoretically grounded approach to novelty detection. Unlike SoftMax-based methods, energy scores are not normalized across classes, making them more sensitive to distributional shifts and unfamiliar modulation schemes.

A regularization technique that improves out-of-distribution detection by training the model with an auxiliary dataset of diverse outlier examples. By exposing the network to a wide variety of non-target signals during training, the model learns to tighten its decision boundary around known classes. This forces the feature extractor to produce more conservative embeddings, reducing the open space risk without requiring modifications to the base architecture.

The post-processing or training technique of aligning a model's predicted probability with the actual likelihood of correctness. A well-calibrated model ensures that a low confidence score reliably indicates an unknown input. Temperature scaling is a simple and effective method that divides output logits by a learned scalar parameter to soften SoftMax probabilities without changing accuracy, making it a lightweight addition to any modulation classifier.

A method for uncertainty quantification that trains multiple neural networks with different random initializations and uses the variance of their predictions as a robust signal for detecting unknown inputs. Each model in the ensemble learns a slightly different functional interpretation of the training data, and their disagreement on unfamiliar modulation types provides a reliable measure of epistemic uncertainty—the model's uncertainty arising from a lack of knowledge.