Open-Set Recognition

The core capability of OSR is the simultaneous classification of known classes and the rejection of unknown classes. Unlike closed-set models that force an input into one of K predefined categories, an OSR model must quantify uncertainty and declare 'I don't know this modulation.' This is achieved by learning a decision boundary that encloses known class manifolds in feature space, leaving the rest of the space as the open-set region. A robust OSR system minimizes both open-set risk—the probability of misclassifying an unknown as a known—and closed-set risk—the probability of misclassifying one known class as another.

An adversarial learning framework that explicitly models the boundary between known and unknown space. Instead of just learning class centroids, the model learns reciprocal points—representations of the potential open space—for each known class. The classifier is trained to push known samples toward their class centroid while pulling them away from all reciprocal points. During inference, the distance to reciprocal points serves as a direct measure of openness, enabling principled rejection of inputs that lie closer to the 'otherness' representation than to any known class.

An enhancement to OpenMax that uses a Generative Adversarial Network (GAN) to synthesize realistic samples of unknown classes during training. By generating synthetic open-set examples that lie near the boundaries of known class manifolds, the classifier learns a tighter, more informed decision boundary. This addresses a key weakness of standard OpenMax: its reliance on only known-class statistics. G-OpenMax explicitly exposes the model to 'what an unknown might look like,' significantly improving open-set detection at lower signal-to-noise ratios.

OSR is mathematically grounded in Extreme Value Theory, which models the distribution of rare events in the tails of probability distributions. The key insight is that the probability of a sample belonging to a known class decays as a function of its distance from the class's training examples. By fitting an EVT model—specifically a Weibull distribution—to the tail of the distance distribution for each class, OSR algorithms can compute a statistically principled probability that a given input is an outlier. This provides a theoretical basis for setting rejection thresholds.

Modern OSR approaches use deep metric learning objectives like prototypical networks or angular margin losses to structure the embedding space. The goal is to create compact, well-separated clusters for known classes with a large angular margin between them. Unknown samples naturally fall into the interstitial space between clusters or far from any centroid. By measuring cosine similarity or Euclidean distance to learned prototypes, the system can reject inputs that fail to exceed a similarity threshold, making the rejection decision geometrically interpretable.

The fundamental distinction between traditional and open-world classification paradigms. In closed-set AMC, the model assumes all test signals belong to one of the K modulation classes seen during training, forcing a prediction even for unknown waveforms. Open-set recognition relaxes this assumption, requiring the model to both classify known modulations and reject samples from unknown classes not present in the training set. This is critical in electronic warfare, where adversaries constantly deploy novel waveforms.

A formal measure of the risk incurred by labeling a sample from an unknown class as belonging to a known class. Open space risk quantifies the probability that a classifier will incorrectly assign a high-confidence label to a point far from any training data. Minimizing this risk is the central objective of open-set recognition algorithms. Techniques like extreme value theory are used to model the boundary between known feature space and the infinite open space beyond.

The task of identifying input signals whose underlying distribution differs fundamentally from the training data. In AMC, an OOD detector flags signals with modulation schemes, channel conditions, or hardware impairments not represented in the training set. Common approaches include:

• Softmax thresholding: rejecting predictions with low maximum probability
• Energy-based models: assigning a scalar energy score, with OOD samples receiving higher energy
• Mahalanobis distance: measuring distance from class-conditional Gaussian distributions in feature space

A related but distinct paradigm where the model is trained only on known, normal data and must detect any deviation as novel. Unlike open-set recognition, novelty detection does not require classifying the known classes—only flagging anomalies. In spectrum monitoring, a one-class classifier trained on authorized transmission patterns can detect rogue emitters, jamming signals, or unauthorized spectrum usage without needing labeled examples of every possible intrusion type.

The mechanism by which a classifier declines to make a prediction when confidence falls below a calibrated threshold. In cost-sensitive AMC, the penalty for misclassifying an unknown jamming waveform as a known friendly signal far outweighs the cost of abstention. Reject options are implemented via:

• Global threshold: a single confidence cutoff applied uniformly
• Class-conditional thresholds: per-class rejection criteria based on known class compactness
• Chow's rule: optimal rejection that minimizes expected risk given a defined cost matrix