Open Set Recognition

The core mathematical principle of OSR is to bound open space risk—the relative measure of the feature space that is labeled as 'known' but is actually far from any training data. A robust OSR model minimizes this risk by tightly encapsulating known classes, ensuring that the infinite space of unknown emitters is not accidentally classified as a known device. This is achieved by moving beyond simple discriminative boundaries to generative or distance-based models that model the support of each known class.

OSR systems rely on explicit rejection mechanisms to flag unknown inputs. Common techniques include:

• Thresholding on Softmax Probability: Rejecting inputs where the maximum predicted probability falls below a calibrated threshold, though this is often unreliable against out-of-distribution samples.
• OpenMax Layer: A specialized output layer that replaces the standard SoftMax by fitting a Weibull distribution to the distance of a sample from class means, explicitly adding a probability score for the 'unknown' class.
• Distance in Embedding Space: Using the distance to the nearest class prototype or the sparsity of the activation vector in a DNN's latent space to detect novelty.

OSR methodologies are broadly categorized by their underlying modeling philosophy:

• Discriminative Models: Standard deep neural networks adapted with rejection layers (like OpenMax) or trained with auxiliary outlier data. They focus on drawing a boundary but require careful calibration to avoid overconfidence on unknowns.
• Generative Models: Architectures like Variational Autoencoders (VAEs) or Generative Adversarial Networks (GANs) learn the probability distribution of known classes. An unknown sample is detected by its low likelihood under the learned distribution, providing a principled statistical basis for rejection.

While related, these terms are distinct:

• Novelty Detection: A one-class problem. The model is trained only on a single 'normal' class and must detect anything that deviates from it. It does not differentiate between multiple known classes.
• Open Set Recognition: A multi-class problem. The model must accurately classify inputs into one of several known positive classes while simultaneously rejecting inputs from an infinite set of unknown negative classes. OSR requires both intra-class discrimination and inter-class novelty detection.

Standard accuracy is insufficient for evaluating OSR. Key metrics include:

• Open Set Classification Rate (OSCR): A curve that plots the correct classification rate of knowns against the false positive rate for unknowns as the rejection threshold varies.
• Area Under the ROC Curve (AUROC): Measures the model's ability to separate known and unknown samples across all thresholds.
• F1-Score in Open Set: The harmonic mean of precision and recall, calculated only on the subset of samples that are both known and correctly classified, penalizing both misclassification and false acceptance of unknowns.

In adversarial device spoofing detection, OSR is critical. A closed-set model trained on 50 authorized transmitters will fail catastrophically when presented with a 51st, unknown spoofing device, confidently mapping it to one of the known 50. An OSR model, conversely, will isolate the unknown emitter's signature in an open space region and correctly flag it as an intruder, triggering an alert for a zero-day impersonation attack without needing prior samples of the specific spoofing hardware.

Military and regulatory spectrum monitoring systems use OSR to identify known threat emitters while flagging never-before-seen waveforms for analyst review. Unlike closed-set classifiers that force every signal into a predefined category, OSR models generate an open-set probability score that triggers alerts when a transmission falls outside the training distribution.

• Detects novel radar modes and previously unobserved communication protocols
• Prevents misclassification of experimental or adversarial waveforms as benign
• Reduces analyst fatigue by prioritizing truly unknown signals

Cognitive radios operating in shared spectrum bands encounter unpredictable interference sources and unlicensed transmitters. OSR enables these systems to distinguish between known cooperative nodes and unknown rogue transmitters that could cause harmful interference.

• Rejects unauthorized secondary users attempting to access licensed spectrum
• Identifies jamming signals that do not match known interference patterns
• Enables spectrum etiquette enforcement without requiring prior knowledge of every possible emitter

Enterprise IoT networks face constant pressure from unauthorized devices attempting to connect. OSR-based physical layer authentication accepts enrolled devices while rejecting both known spoofing attacks and completely novel impersonation techniques.

• Blocks zero-day spoofing hardware that closed-set models would misclassify as legitimate
• Maintains security as new device models enter the ecosystem without retraining on every variant
• Supports scalable zero-trust architectures where unknown equals untrusted

Electronic warfare (EW) systems maintain libraries of known hostile emitter signatures. OSR prevents dangerous misclassification when an adversary deploys a modified or previously undocumented system. The model outputs a novelty detection score rather than forcing a match to the closest known threat.

• Flags parametrically modified variants of known radar systems
• Detects completely new electronic attack waveforms in real-time
• Triggers automated signal collection for subsequent forensic analysis and library updates

Manufacturing environments rely on deterministic wireless links for robotic control and sensor telemetry. OSR monitors the electromagnetic environment to detect unknown interference sources and unauthorized transmitters that could disrupt operations.

• Identifies faulty transmitters exhibiting anomalous impairment signatures
• Detects unauthorized wireless devices introduced into secure production areas
• Distinguishes between normal environmental variation and genuinely novel threats

Satellite uplinks are vulnerable to spoofing and command injection attacks. OSR authenticates known ground station transmitters by their RF fingerprint while rejecting any unknown emitter attempting to communicate with the spacecraft.

• Prevents unauthorized command uplink attempts from unregistered ground stations
• Detects mobile spoofing platforms that do not match enrolled transmitter profiles
• Operates effectively with minimal training examples per authorized ground station

A one-class classification paradigm that models the distribution of known normal data and identifies any deviation as novel. Unlike open set recognition, traditional novelty detection does not distinguish between multiple known classes.

• Training Regime: Trained exclusively on positive (in-distribution) samples with no exposure to outliers.
• Common Algorithms: One-Class SVM, Isolation Forest, and deep autoencoders with reconstruction error thresholds.
• OSR Integration: Often used as the rejection component within a broader open set architecture to flag unknown emitters.

The standard supervised learning paradigm where the model assumes all possible classes are known during training. Every input at inference time is forcibly mapped to one of the pre-defined categories, making it fundamentally vulnerable to spoofing.

• Core Limitation: A closed set model will confidently misclassify an unknown adversarial device as a legitimate emitter with high probability.
• Softmax Vulnerability: The softmax function normalizes outputs to sum to 1, eliminating any mechanism to express uncertainty about unknown classes.
• Contrast with OSR: Open set recognition explicitly adds a reject option to the classifier's decision space.

A training regularization technique that exposes a model to auxiliary outlier datasets during training to force the network to learn more conservative decision boundaries for unknown classes.

• Mechanism: Adds a loss term that pushes outlier samples toward a uniform distribution over known classes or a dedicated background class.
• Data Requirements: Requires a carefully curated set of out-of-distribution examples that are representative of potential unknowns.
• Benefit: Dramatically improves open set performance without requiring examples of every possible unknown emitter during training.

A metric that characterizes the dimensional properties of the data subspace surrounding a sample point. Adversarial and out-of-distribution samples often reside in regions with anomalously high intrinsic dimensionality.

• Mathematical Basis: Estimates the rate of growth in the number of data points as the distance from a reference sample increases.
• Detection Application: A sharp increase in LID indicates the sample lies off the training data manifold, enabling robust open set rejection.
• Advantage: Provides a model-agnostic detection statistic that does not require retraining the classifier.

A self-supervised representation learning approach that structures the embedding space to maximize separation between known classes while creating a compact, unified region for all potential unknowns.

• Training Objective: Pulls augmented views of the same sample together while pushing all other samples apart using a contrastive loss like InfoNCE.
• Open Set Advantage: Learns features that generalize to unseen classes because the objective does not rely on fixed class labels.
• Rejection Strategy: Unknown samples are identified by their distance from any known class prototype in the learned embedding space.