Out-of-Distribution Detection

The formal classification paradigm where a model must simultaneously accurately classify known classes and reject unknown classes not seen during training. Unlike closed-set problems that force a prediction into one of K classes, open set recognition introduces a reject option for OOD inputs.

• Open Space Risk: The measurable risk of labeling an unknown input as known, bounded by the ratio of open space to total measurable space
• Compact Abatement Principle: Known class representations must be bounded and compact to minimize open space risk
• Key distinction from novelty detection: Open set recognition requires both known class discrimination and unknown rejection simultaneously

A baseline OOD detection method that uses the maximum softmax probability as a confidence score. Inputs with max probability below a calibrated threshold are flagged as OOD.

• Temperature scaling can calibrate softmax outputs to better reflect true posterior probabilities
• Overconfidence problem: Deep networks often produce high-confidence predictions on OOD inputs, making raw softmax unreliable
• Expected Calibration Error (ECE) measures the gap between confidence and accuracy, directly impacting threshold reliability
• In RF applications, noise-like OOD signals can trigger high softmax scores on visually similar in-distribution modulation classes

A parametric OOD detection method that fits a class-conditional Gaussian distribution to the feature representations of each known class. The minimum Mahalanobis distance from an input's feature vector to any class centroid serves as the OOD score.

• Operates on penultimate layer features rather than logits, capturing richer representational structure
• Requires computing per-class mean vectors and covariance matrices from training data
• Shared covariance assumption can improve robustness when per-class samples are limited
• Particularly effective for RF signals where known modulation schemes form distinct clusters in feature space with well-defined covariance structures

Uses the Helmholtz free energy formulation of a discriminative classifier as an OOD score. The energy score E(x) = -T · log Σᵢ exp(fᵢ(x)/T) is lower for in-distribution inputs and higher for OOD inputs, providing a theoretically grounded alternative to softmax.

• Temperature parameter T controls the smoothness of the energy landscape
• Aligns with energy-based models where in-distribution data occupies low-energy basins
• Outperforms softmax confidence because the energy score is not normalized across classes, preserving magnitude information
• Computationally efficient: requires only a single forward pass with no auxiliary models or gradient computations

Out-of-DIstribution detector for Neural networks that combines temperature scaling and input perturbation to amplify the separability between in-distribution and OOD softmax scores.

• Temperature scaling: Raises softmax temperature to sharpen or smooth probability distributions
• Input pre-processing: Adds small adversarial perturbations calculated as the gradient of the softmax score with respect to the input
• Perturbation magnitude ε is a calibration hyperparameter tuned on a validation OOD set
• The two-step process widens the gap between in-distribution and OOD maximum softmax values, making threshold selection more robust

A density estimation approach that models the feature space distribution of in-distribution data using a Gaussian Mixture Model (GMM). The log-likelihood of a test sample under the GMM serves as the OOD score.

• Captures multi-modal distributions common in RF data where a single modulation class may have multiple operational modes
• Bayesian Information Criterion (BIC) guides selection of the optimal number of mixture components
• Unlike distance-based methods, GMMs model the full density including low-probability regions between clusters
• Particularly useful when known RF classes exhibit sub-populations due to varying hardware impairments or channel conditions