Novelty Detection

Novelty detection operates as a one-class classification problem where the decision boundary is sculpted solely from positive (normal) samples. The model learns a compact description of the target class and treats any point outside this boundary as novel. This is fundamentally different from binary classification because counterexamples are absent during training. Techniques like One-Class SVM and Deep SVDD implement this by finding the minimal-volume hypersphere or hyperplane that encapsulates the training data in a high-dimensional feature space.

The core statistical mechanism involves estimating the probability density function (PDF) of the normal data. Regions of high density define normality; low-density regions signal novelty. Parametric methods fit a Gaussian Mixture Model, while non-parametric approaches like Kernel Density Estimation (KDE) place a kernel on each training point. The model flags a sample as novel if its estimated density falls below a calibrated threshold, effectively rejecting points in low-support regions of the feature space.

Autoencoders and their variants learn to compress and reconstruct normal data with minimal error. The assumption is that a network trained exclusively on normal patterns will produce a high reconstruction error when fed a novel sample it cannot faithfully encode. The residual between the input and its reconstruction serves as the anomaly score. Variational Autoencoders (VAEs) extend this by modeling the latent space probabilistically, using the reconstruction probability rather than raw error for more robust novelty scoring.

This approach computes the distance between a test sample and a representative prototype of the normal class in a learned metric space. Prototypical Networks and Deep SVDD minimize the volume of a hypersphere centered on a prototype that encloses all normal embeddings. The Mahalanobis distance provides a more sophisticated metric by accounting for the covariance structure of the normal data distribution, measuring distance in units of standard deviation along each principal component axis.

The Isolation Forest algorithm exploits the fact that anomalies are few and different. It builds an ensemble of random trees by recursively partitioning the feature space with random splits. The key insight: a novel point requires fewer random partitions to be isolated than a normal point buried deep in a dense cluster. The average path length across the ensemble serves as the novelty score, with shorter paths indicating higher novelty. This method is parameter-free regarding distributional assumptions and scales efficiently to high-dimensional data.

Setting the decision threshold is critical. Extreme Value Theory (EVT) models the tail of the score distribution to calibrate thresholds with statistical rigor. Instead of assuming a Gaussian distribution, EVT fits a Generalized Pareto Distribution to the extreme scores, enabling precise control over the false positive rate. This is the foundation of the OpenMax algorithm, which uses a Weibull distribution to model the distance of a sample from its class mean, providing a calibrated probability of novelty.

The task of identifying input samples that differ significantly from the training data distribution, triggering a rejection mechanism. In RF fingerprinting, an OOD detector flags signals from transmitters whose hardware impairments fall outside the learned manifold.

• SoftMax thresholding: Reject samples with low maximum class probability
• Energy-based models: Assign high energy scores to OOD inputs
• Mahalanobis distance: Measure deviation from class-conditional Gaussians in feature space

An algorithm that replaces the standard SoftMax layer with a mechanism calibrated using Extreme Value Theory (EVT). It fits a Weibull distribution to the distance between a sample and its class mean, estimating the probability that the input belongs to an unknown class.

• Reduces open space risk by explicitly modeling the unknown
• Enables calibrated rejection of never-before-seen emitter types
• Foundational work by Bendale and Boult (2016)

Deep Support Vector Data Description trains a neural network to map normal data into a minimal-volume hypersphere. Points falling outside this boundary are classified as anomalies or novelties.

• One-class classification: Requires only normal samples for training
• The center of the hypersphere is learned jointly with the feature representation
• Extension Deep SAD incorporates a small amount of labeled anomalies to refine the boundary

Distinguishing between two types of uncertainty is critical for novelty detection in dynamic spectrum environments:

• Epistemic uncertainty: Reducible model uncertainty from lack of knowledge. High for inputs far from training data — a key signal for novelty
• Aleatoric uncertainty: Irreducible noise inherent in the data, such as I/Q imbalance measurement variance or channel fading
• Monte Carlo Dropout and Evidential Deep Learning estimate both quantities for robust rejection

A distribution-free framework that produces prediction sets with a guaranteed marginal coverage probability. Rather than a single class label, the model outputs a set of plausible classes.

• If the set is empty, the sample is flagged as novel
• Provides rigorous statistical guarantees without assuming any distribution
• Useful for safety-critical emitter authentication where false acceptance is unacceptable

A dynamic paradigm extending beyond novelty detection. The model must:

1. Detect unknown emitter classes in real-time
2. Incrementally learn new classes from these unknowns
3. Retain knowledge of previous classes without catastrophic forgetting

This contrasts with static novelty detection by closing the loop between discovery and model adaptation, essential for long-term spectrum monitoring deployments.