Out-of-Distribution Detection

The most straightforward baseline for OOD detection relies on the model's output confidence. If the maximum softmax probability is below a calibrated threshold, the input is flagged as OOD.

• Mechanism: Assumes in-distribution (ID) samples produce high-confidence predictions, while OOD samples yield diffuse, low-confidence probability vectors.
• RF Limitation: Modern deep networks are often poorly calibrated and can produce overconfident predictions on nonsense inputs. In RF, a high-power jammer with a slightly unfamiliar pattern might still trigger a high softmax score for a known modulation class.
• Enhancement: Temperature scaling and energy-based models refine this baseline by reshaping the logit landscape to separate ID and OOD densities more effectively.

Rather than trusting the final classification layer, distance-based methods operate on the penultimate feature layer (embeddings). They measure the distance of a test sample's embedding to the nearest class prototype or training cluster.

• Mahalanobis Distance: Computes the distance to the nearest class-conditional Gaussian distribution, accounting for feature covariance. This is highly effective for detecting subtle hardware impairments in RF fingerprinting.
• Prototypical Networks: In few-shot RF settings, a sample is classified as OOD if its embedding is too far from any learned prototype in Euclidean space.
• Advantage: These methods are architecture-agnostic and provide a more geometrically grounded uncertainty estimate than softmax scores.

Energy-based models reframe OOD detection by learning a scalar energy function E(x) that assigns low energy to in-distribution data and high energy to OOD data. This is a theoretically grounded alternative to the softmax probability.

• Helmholtz Free Energy: The logits of a discriminative classifier can be reinterpreted as defining an energy landscape. The free energy score aligns the model's confidence with the data density.
• RF Application: EBMs are particularly useful for spectrum sensing networks where the model must reject unknown interference patterns. Instead of forcing a classification, the system simply rejects inputs above an energy threshold.
• Training: Often involves contrastive divergence or noise-contrastive estimation to explicitly carve out low-energy valleys around the training data.

These methods measure how a model reacts to small changes in the input. OOD samples often reside in regions of the input space where the model's decision boundary is unstable.

• ODIN (Out-of-DIstribution detector for Neural networks): Uses temperature scaling and adds small, controlled perturbations to the input to widen the gap in softmax scores between ID and OOD samples.
• GradNorm: Exploits the observation that the gradient norm of the KL divergence between the softmax output and a uniform distribution is typically much higher for ID samples than OOD samples.
• RF Relevance: In automatic modulation classification, a genuine but unknown modulation scheme will exhibit a high GradNorm score, signaling that the model is "surprised" and actively trying to fit the input into a known class.

Generative models like autoencoders or GANs are trained exclusively on in-distribution data. At test time, they will accurately reconstruct ID samples but fail to reconstruct OOD samples, resulting in a high reconstruction error.

• Autoencoder Threshold: A simple mean squared error (MSE) between the input IQ sequence and its reconstruction serves as the OOD score.
• Likelihood Regret: A more sophisticated metric using normalizing flows or VAEs that measures the discrepancy between a generic model and a fine-tuned ID model.
• RF Use Case: This is the gold standard for detecting novel emitters. A denoising autoencoder trained on known friendly emitters will fail to reconstruct a rogue emitter's unique hardware signature, immediately flagging it as anomalous.

It is crucial to distinguish between related but distinct tasks. OOD detection is a binary gate (ID vs. not-ID), while open-set recognition (OSR) must simultaneously classify known classes and reject unknowns.

• Open-Set Recognition: Requires a model to maintain a tight decision boundary around each known class while leaving the rest of the space as "unknown." This is often implemented with OpenMax layers that replace softmax with a Weibull-calibrated rejection mechanism.
• Outlier Detection: Focuses purely on finding anomalies in unlabeled data, often using one-class SVMs or isolation forests, without the need to classify known signals.
• RF Context: A spectrum monitor performing dynamic spectrum awareness uses OOD to ignore known signals and OSR to categorize them while simultaneously logging novel rogue transmissions for analyst review.

A self-supervised learning method that learns representations by predicting future latent representations from past ones using a probabilistic contrastive loss. In RF applications, CPC trains on unlabeled IQ streams to capture the underlying structure of legitimate signals, making deviations from this structure—out-of-distribution samples—immediately detectable as anomalies in the latent space.

A meta-learning algorithm for few-shot classification that computes a prototype representation for each known class as the mean of its support set embeddings. For OOD detection, query samples that fall far from all known prototypes in the embedding space are flagged as novel or unknown. This distance-based approach provides a natural, interpretable rejection mechanism for unrecognized emitters.

The ability of a model trained on specific RF environments to perform accurately on unseen target domains with different channel conditions or hardware impairments without any adaptation. Strong domain generalization is critical for OOD detection because a model must distinguish between covariate shift (known signal, new channel) and semantic shift (truly unknown signal class).

A failure mode in self-supervised learning where the encoder produces a constant or non-informative output for all inputs. A collapsed representation is catastrophic for OOD detection because it maps both in-distribution and out-of-distribution samples to the same region, eliminating any ability to differentiate known from unknown signals. Techniques like variance regularization and covariance regularization explicitly prevent this.

A semi-supervised technique where a model trained on labeled data generates artificial labels for unlabeled data, and the model is retrained using both real and high-confidence pseudo-labeled examples. In open-world RF monitoring, samples that receive low-confidence pseudo-labels across all known classes are strong candidates for out-of-distribution detection, serving as a practical rejection heuristic.

A semi-supervised learning principle that enforces a model to produce similar predictions for an unlabeled data point and its perturbed or augmented versions. For OOD detection, this principle is inverted: out-of-distribution samples often exhibit high prediction inconsistency under augmentation, providing a measurable signal that the input lies outside the training manifold.