Open Set Recognition

Traditional closed-set classifiers operate under the rigid assumption that all test samples belong to one of the K known training classes, forcing a prediction even for alien signals. Open Set Recognition relaxes this assumption, requiring the model to perform novelty detection by assigning a 'none of the above' label to unknown modulations. This prevents high-confidence misclassifications of new waveforms.

A formal definition of OSR involves bounding the open space risk—the relative measure of the feature space labeled as 'unknown' compared to the space labeled as 'known'. Effective OSR models minimize this risk by tightly encapsulating known classes. Techniques include:

• Extreme Value Theory (EVT) for modeling the tails of class distributions.
• Discriminative models that carve out compact, bounded regions for each known modulation.

OSR systems rely on a rejection layer that evaluates the confidence of the primary classifier. Common approaches include:

• Softmax Thresholding: Rejecting predictions where the maximum posterior probability falls below a calibrated threshold.
• OpenMax: Replacing the standard softmax layer with one that estimates the probability of an input belonging to an unknown class by fitting Weibull distributions to the penultimate layer's activation vectors.
• Energy-Based Models: Using Helmholtz free energy scores as a discriminative metric; known classes exhibit lower energy than unknowns.

The effectiveness of OSR is heavily dependent on the geometry of the learned embedding space. Models are trained to create compact, well-separated clusters for known classes while maximizing the distance between clusters and the origin or a learned boundary. Techniques like prototypical networks and angular margin penalties enforce intra-class compactness and inter-class separability, leaving a large, unoccupied region in the latent space for unknown signals to be detected.

Autoencoders and Generative Adversarial Networks (GANs) provide an alternative OSR framework by learning the manifold of known signal types. The reconstruction error of an autoencoder serves as a novelty score; an unknown modulation will yield a high reconstruction error because it lies off the learned manifold. Similarly, GAN-based anomaly detection uses the discriminator's inability to accurately represent a novel sample as a signal for rejection.

Standard accuracy is insufficient for OSR. Performance is measured by the trade-off between open-set rejection and closed-set accuracy. Key metrics include:

• AUROC: Area Under the Receiver Operating Characteristic curve for the binary task of known vs. unknown detection.
• F1-Score at a fixed known-class accuracy: Measuring the balance between correctly identifying known modulations and flagging unknowns.
• Open Set Classification Rate (OSCR): A metric that evaluates the correct classification rate as a function of the rejection rate.

The fundamental distinction in classification paradigms. In a closed set, the test classes are identical to the training classes. In an open set, the model encounters unknown classes during inference that were absent from training. A standard Softmax classifier forces a decision among known classes, making it unsuitable for open worlds where rejecting an anomaly is critical.

A drop-in replacement for the standard Softmax layer that enables deep networks to estimate the probability of an input belonging to an unknown class. It works by fitting a Weibull distribution to the penultimate layer's activation vectors for each known class and recalibrating the output scores to include a dedicated 'unknown' pseudo-class.

A closely related field focused on identifying inputs that are semantically different from the training distribution. Key techniques include:

• MSP Baseline: Using the Maximum Softmax Probability as a confidence score.
• ODIN: Adding small gradient-based perturbations and temperature scaling to separate in-distribution and out-of-distribution softmax scores.
• Mahalanobis Distance: Fitting class-conditional Gaussian distributions to intermediate features for a calibrated confidence metric.

A decision-theoretic mechanism that allows a classifier to abstain from making a prediction when its confidence is below a defined threshold. In modulation recognition, this prevents a cognitive radio from misidentifying a novel jamming waveform as a benign QPSK signal. The trade-off is managed by a risk-coverage curve, balancing the rate of rejection against classification accuracy on accepted samples.

A metric-based few-shot learning architecture that computes an embedding prototype for each class by averaging the latent representations of its support examples. For open set recognition, the distance of a query sample to the nearest class prototype is used as a rejection metric. If the distance exceeds a calibrated threshold, the signal is flagged as unknown, enabling classification without retraining on novel emitters.

The statistical foundation underpinning many open set algorithms. EVT models the behavior of the tails of a distribution, allowing systems to estimate the probability of observing an extreme activation value. Weibull distributions are fitted to the distance between correct class activations and their mean, providing a rigorous probabilistic framework for bounding the open space risk rather than relying on arbitrary confidence thresholds.