Out-of-Distribution (OOD) Detection

OOD detection must distinguish between two critical scenarios. Covariate shift refers to changes in the input distribution that do not alter the fundamental class (e.g., a known radar signal with a lower signal-to-noise ratio due to distance). Semantic novelty is the true target: a completely new modulation scheme or emitter type that belongs to an unknown class. A robust detector ignores the former while flagging the latter, preventing alert fatigue in spectrum monitoring operations.

These methods rely on the geometric properties of a learned feature space. After training a deep network on normal I/Q samples, the model maps new inputs to a compact embedding. Key techniques include:

• Mahalanobis Distance: Measures the distance from class-conditional Gaussian centroids, accounting for feature covariance.
• Deep SVDD: Trains the network to enclose normal data within a minimal-volume hypersphere; points outside are anomalies.
• K-NN in Feature Space: Flags inputs whose nearest neighbors in the training set are beyond a threshold distance.

These methods explicitly model the probability distribution of the training data and flag low-likelihood regions. Gaussian Mixture Models (GMMs) fit a weighted sum of Gaussians to the latent space, while Normalizing Flows learn a bijective mapping to a simple base distribution for exact likelihood computation. Variational Autoencoders (VAEs) provide a probabilistic alternative, using the Evidence Lower Bound (ELBO) as an anomaly score. A low likelihood indicates the sample originates from outside the training manifold.

Autoencoders trained exclusively on normal spectrum data learn to compress and reconstruct it with minimal error. The core assumption is that reconstruction error serves as a proxy for novelty. A standard autoencoder or a temporal variant like an LSTM Autoencoder will fail to accurately reconstruct an anomalous signal it has never seen, producing a high mean squared error. This technique is effective for detecting transient interference or jamming pulses that deviate from the learned background noise and communication patterns.

This approach uses the softmax probabilities of a classifier directly. A model trained to identify known modulation types (BPSK, QPSK, 16QAM) will typically produce a low maximum softmax probability or a high entropy output vector when fed an unknown signal. Energy-based models refine this by mapping logits to a free energy score. ODIN (Out-of-Distribution detector for Neural networks) enhances this separation by applying temperature scaling and small input perturbations to amplify the gap between in-distribution and OOD scores.

OOD detection is formalized as Open Set Recognition (OSR). Unlike closed-set classification, OSR must balance two risks: misclassifying an unknown attack signal as a known friendly emitter, and rejecting a noisy but valid known signal. The decision boundary must include a 'none of the above' region. Evaluation metrics like AUROC and FPR at 95% TPR are standard, but spectrum-specific metrics must also account for the temporal persistence of an anomaly to avoid flickering detections on a waterfall display.

The fundamental anomaly score in many neural network-based detectors. An autoencoder is trained exclusively on normal spectrum data to minimize the difference between its input and its reconstruction. At inference time, an OOD signal that the model has never seen will produce a high reconstruction error, as the model's compressed latent representation cannot accurately regenerate the novel input. This metric is the primary trigger for flagging a potential rogue emitter.

A multivariate statistical measure that quantifies a sample's distance from a distribution's mean, crucially accounting for the covariance between features. Unlike Euclidean distance, it is scale-invariant and considers correlations in the data. In OOD detection, a feature vector extracted from a received signal is scored against the learned distribution of normal signal features. A high Mahalanobis distance indicates the signal is statistically anomalous and likely out-of-distribution.

Deep Support Vector Data Description is a neural one-class classification method. The objective is to train a neural network to map all normal training data into a minimal hypersphere in a high-dimensional feature space. The network's parameters are optimized to minimize the volume of this sphere. During deployment, any new signal that maps to a point outside this learned boundary is classified as an anomaly or OOD sample, making it highly effective for open-set spectrum recognition.

A classification paradigm that moves beyond traditional closed-set assumptions. In spectrum monitoring, a model must not only correctly identify known modulation schemes (e.g., QPSK, 16QAM) but also detect and reject unknown, novel signal types. OOD detection is the core mechanism enabling this. An open set recognizer explicitly models the decision boundary between known classes and the rest of the open space, preventing a rogue LPI waveform from being confidently misclassified as a known friendly signal.

The process of identifying when the statistical properties of the RF environment change over time. A new emitter, a shift in propagation conditions, or a change in background interference can cause the data distribution to drift from the training set. OOD detection systems must be paired with drift detection to distinguish between a transient anomaly and a permanent shift in normality. Without this, a model's baseline for 'normal' becomes stale, increasing false positive rates.

A learned, low-dimensional vector representation of raw I/Q samples or spectral features. Instead of applying OOD detection to high-dimensional raw data, a neural network is first trained to map inputs to a compact embedding space where semantically similar signals cluster together. OOD scoring is then performed in this latent space, where distances are more meaningful. A signal that maps to a sparse, low-density region of the embedding space is a strong candidate for being out-of-distribution.