Out-of-Distribution (OOD) Signal Detection

OOD detection relies on modeling the probability density of the training data. Gaussian Mixture Models or Kernel Density Estimation map the latent space of known signals. An input is flagged as OOD if its likelihood score falls below a calibrated threshold, indicating it resides in a low-density region of the learned manifold. This prevents the model from confidently misclassifying a novel jamming waveform as a known modulation scheme.

Unlike traditional closed-set classifiers that force every input into a known category, OOD systems implement Open-Set Recognition. The model explicitly rejects unknown classes rather than mapping them to the nearest incorrect label.

• Closed-Set: Assumes all test classes were seen during training.
• Open-Set: Introduces an 'unknown' or 'reject' class for novel interference. This is essential in electronic warfare where adversaries constantly deploy new jamming strategies.

A common OOD mechanism measures the distance of a new input's feature vector to the nearest training sample or class centroid in the embedding space.

• Mahalanobis Distance: Accounts for the covariance structure of the training distribution, providing a more robust metric than Euclidean distance.
• Deep Nearest Neighbors: Uses the distance to the k-th nearest neighbor in the penultimate layer of a neural network as an OOD score. If the distance exceeds a learned radius, the signal is rejected as anomalous.

A baseline OOD method uses the maximum softmax probability (MSP) as a confidence score. The intuition is that a model will produce a low maximum probability for inputs far from its training data. However, modern neural networks are often overconfident on OOD inputs, assigning high probabilities to nonsensical classifications. Advanced techniques like Temperature Scaling and Energy-Based Models recalibrate these scores to create a sharper separation between in-distribution and OOD signals.

Self-supervised contrastive learning is used to pre-train feature extractors that naturally separate OOD data. By pulling representations of augmented views of the same signal together and pushing different signals apart, the model learns a latent space where novel interference types cluster distinctly from the training distribution. This pre-training significantly improves the recall of OOD detectors without requiring any anomalous samples during training.

OOD detection is tightly coupled with adversarial robustness. An intelligent jammer may craft evasion attacks—subtle perturbations that push a malicious waveform just inside the decision boundary of a known class. Robust OOD detectors use adversarial training or Local Intrinsic Dimensionality (LID) analysis to identify these manipulated inputs. LID measures the rate of growth in the number of data points as the distance from the sample increases, revealing the adversarial subspace.

The classification paradigm where a model must not only identify known signal types but also detect and flag previously unseen or unknown interference patterns. Unlike closed-set classifiers that force a prediction into a known category, open-set recognition explicitly models the decision boundary to reject anomalies. This is the direct parent concept of OOD detection, ensuring a system can say 'I don't know what this is' rather than misclassifying a novel jamming waveform as a benign signal.

The process of using machine learning models to identify intentional jamming or spoofing signals designed to evade traditional detection systems. Adversarial signals are often crafted to sit precisely on the boundary of the training distribution, making them a critical test case for OOD detectors. A robust OOD system must distinguish between naturally occurring novel signals and those maliciously engineered to mimic in-distribution data through subtle waveform manipulation.

The categorization of unusual or unauthorized transmissions within a monitored frequency band using unsupervised or semi-supervised learning models. While OOD detection focuses on the input data distribution, anomaly classification operates on the spectral occupancy patterns. These two concepts are deeply intertwined: an OOD signal input often manifests as a spectrum anomaly, and anomaly detectors frequently rely on reconstruction error or density estimation—the same mathematical foundations used in OOD scoring functions.

A neural network architecture that directly processes in-phase and quadrature (IQ) data as complex numbers, preserving the phase relationships critical for RF classification. Standard real-valued networks often fail to capture the intricate geometric properties of signal constellations. For OOD detection, CVNNs provide a more faithful representation of the training manifold, enabling more precise measurement of the Mahalanobis distance or feature-space density that separates in-distribution signals from out-of-distribution outliers.

The application of feature attribution methods like SHAP, LIME, or saliency maps to make the decisions of complex RF classification models interpretable to human analysts. When an OOD detector flags a signal, the immediate operational question is 'why?' XAI techniques decompose the OOD score by highlighting which specific time-frequency regions or statistical features triggered the rejection. This is essential for building operator trust and for debugging whether the detector is responding to genuine novelty or a spurious artifact.

A transfer learning technique that aligns feature distributions between different hardware receivers, environmental conditions, or frequency bands to maintain classification accuracy without manual recalibration. A common failure mode for OOD detectors is flagging signals as novel simply because they were captured on a different receiver front-end. Domain adaptation mitigates this by learning invariant representations, ensuring the OOD boundary is defined by signal semantics rather than hardware-specific artifacts.