One-Class SVM

Projects input features into a high-dimensional kernel space (typically using a Radial Basis Function (RBF) kernel) where the algorithm finds a hyperplane that maximally separates the training data from the origin. The nu parameter controls the fraction of training samples allowed to fall outside the boundary, directly setting the model's tolerance to outliers in the known modulation set. This creates a tight, non-linear envelope around the target class.

For each new IQ sample or feature vector, the model computes a decision function score. A positive score indicates the point lies within the learned boundary (known modulation), while a negative score flags it as novel or anomalous. The magnitude of the score reflects the distance from the boundary, providing a continuous measure of novelty confidence rather than a binary label. This is critical for setting adjustable detection thresholds in spectrum monitoring.

Unlike multi-class classifiers that require labeled examples of all possible modulations, One-Class SVM trains exclusively on positive examples of the known signal type. This makes it ideal for open set recognition where unknown modulation schemes are, by definition, unavailable during training. The model builds a model of normality from only the target class, avoiding the closed-set assumption entirely.

The final decision boundary is defined only by a subset of training points called support vectors—samples that lie on or within the margin. This sparsity makes inference computationally efficient, as only the kernel evaluation against support vectors is needed for each new prediction. For edge-deployed spectrum sensors, this reduces memory footprint and latency compared to dense neural alternatives.

Performance hinges on two critical hyperparameters: nu (upper bound on outlier fraction, lower bound on support vectors) and gamma (RBF kernel width). A small gamma produces a smooth, generalized boundary that may miss subtle anomalies. A large gamma creates a tight, wiggly boundary that overfits to noise. Cross-validation on held-out known samples is essential for tuning these values to the specific signal-to-noise regime.

While effective on engineered features like higher-order cumulants or cyclostationary signatures, One-Class SVM struggles with raw, high-dimensional IQ samples. The curse of dimensionality degrades kernel distance metrics, and the model lacks the hierarchical feature extraction capability of deep learning. For raw waveform novelty detection, it is often outperformed by autoencoder-based anomaly detection or Deep SVDD.

A neural network trained to reconstruct normal training data. During inference, the reconstruction error serves as a novelty score; unknown modulation schemes produce high error as they fall outside the learned manifold.

• Learns a compressed latent representation of known signals
• Flags inputs exceeding a statistically derived error threshold
• Sensitive to architectural choices like bottleneck size

A statistical framework for modeling the tail behavior of distributions. In open set recognition, EVT fits a Weibull distribution to the distance of correct classifications from their class mean, enabling calibrated rejection of unknown modulations.

• Forms the mathematical basis for the OpenMax layer
• Models the probability of extreme events beyond observed data
• Avoids assumptions about the overall distribution shape

A distance metric that accounts for the covariance structure of a class distribution. It measures how many standard deviations a point is from the mean of a distribution, providing a statistically informed score for out-of-distribution detection.

• Reduces to Euclidean distance when features are uncorrelated
• Requires estimating the class-conditional covariance matrix
• More robust to feature scaling than Euclidean distance

A regularization technique that improves novelty detection by training the model with an auxiliary dataset of diverse outlier examples. This forces the network to learn a tighter decision boundary around known classes.

• Requires curating a representative set of non-target signals
• Teaches the model to produce uniform, high-entropy outputs for outliers
• Significantly boosts AUROC for out-of-distribution detection

A class of models that learn an energy function assigning low energy to in-distribution data and high energy to out-of-distribution data. The Helmholtz free energy serves as a discriminative score for novelty.

• Aligns with the physical concept of lowest energy states
• Can be derived from any pre-trained classifier's logits
• Avoids the overconfidence problem of SoftMax probabilities