Open Set Recognition

The fundamental capability of an OSR system is to simultaneously perform closed-set classification on known classes while executing out-of-distribution (OOD) detection for unknown inputs. Unlike standard softmax classifiers that force a prediction into one of K classes, OSR models must quantify epistemic uncertainty to recognize when an input lies outside the learned manifold. This is typically achieved by thresholding a confidence score or distance metric in an embedding space, rejecting samples that fall beyond a learned boundary.

OSR formalizes the concept of open space risk—the measurable danger of labeling an unknown input as a known class. The objective is to bound this risk by defining a compact abating probability model. In practice, this means the classifier's decision boundary must tightly envelop known class regions while leaving vast, unbounded open space labeled as 'unknown'. Techniques like Extreme Value Theory (EVT) are often applied to model the tails of class distributions, enabling statistically rigorous rejection of outliers.

OSR systems implement explicit rejection strategies rather than relying on implicit softmax probabilities. Common mechanisms include:

• Threshold-based rejection: Rejecting samples where the maximum predicted probability falls below a calibrated threshold
• Distance-based rejection: Using cosine similarity or Euclidean distance in an embedding space to measure proximity to known class prototypes
• Background class modeling: Training an explicit 'background' or 'unknown' class using auxiliary outlier data
• Energy-based models: Computing an energy score from a model's logits, where high energy indicates an out-of-distribution sample

A critical distinction exists between Open Set Recognition and Open World Recognition. OSR focuses on the static problem of identifying unknowns at inference time. Open World Recognition extends this by requiring the system to incrementally learn newly discovered classes without catastrophic forgetting of previously learned ones. This demands integration with continual learning algorithms like Elastic Weight Consolidation (EWC) or memory replay buffers, enabling the model to evolve its taxonomy as novel emitters or devices are enrolled.

Standard accuracy metrics fail to capture OSR performance. Key evaluation protocols include:

• Open Set Classification Rate (OSCR): Measures the trade-off between correct classification of knowns and correct rejection of unknowns as a function of rejection threshold
• Area Under the ROC Curve (AUROC) for OOD detection: Quantifies how well the model separates known from unknown samples
• False Positive Rate at 95% True Positive Rate (FPR95): Measures the rate of unknown samples incorrectly classified as known when 95% of known samples are correctly identified
• Closed-set accuracy on known classes combined with open-set F1-score provides a holistic view of joint performance

In Radio Frequency Fingerprinting, OSR is essential for Open Set Emitter Recognition. A deployed fingerprinting system must authenticate enrolled devices while flagging any previously unseen transmitter as a potential rogue or spoofed device. This directly impacts False Acceptance Rate (FAR) in Physical Layer Authentication systems. When combined with Few-Shot Device Enrollment, an OSR model can rapidly incorporate new authorized devices into its known set while maintaining a robust boundary against the infinite space of unknown emitters in a dynamic electromagnetic environment.

The fundamental distinction in recognition paradigms. Closed Set assumes all test classes were seen during training, forcing every input into a known category. Open Set acknowledges the existence of unknown unknowns—classes absent from the training set. In RF fingerprinting, a closed set model would incorrectly classify a new, unauthorized transmitter as a known device, whereas an open set model would flag it as unrecognized, triggering a security alert. This shift from forced-choice to rejection-based classification is essential for zero-trust architectures.

A formalization of the risk incurred by labeling an input from an unknown class as known. Defined mathematically by Scheirer et al., open space risk is proportional to the ratio of the volume of space labeled as 'known' relative to the total measurable space. The goal of any open set algorithm is to minimize this risk by bounding the open space—tightly encapsulating known class regions while leaving the vast remainder of the feature space designated as unknown. This concept provides the theoretical grounding for rejection mechanisms.

A closely related but distinct task. OOD detection identifies inputs that are semantically or statistically different from the training distribution, which may include covariate shifts or samples from unrelated domains. Open Set Recognition is a specific subset of OOD detection focused on novel class rejection within the same overarching domain. For example, an RF model detecting a Wi-Fi signal when trained only on Bluetooth is OOD; detecting a new, unseen Bluetooth device model is open set. Both rely on quantifying prediction uncertainty.

A statistical framework for modeling the behavior of tail-end events, adapted for open set recognition to calibrate rejection thresholds. The Pickands-Balkema-de Haan theorem states that the distribution of values exceeding a sufficiently high threshold converges to a Generalized Pareto Distribution. In practice, EVT is used to model the distance distributions of known class samples from their class centroids, enabling statistically rigorous confidence bounds. The OpenMax algorithm, a seminal work, replaces softmax layers with EVT-calibrated Weibull distributions per class.

A foundational open set classifier that replaces the standard softmax layer with an EVT-based activation. The process:

• Compute activation vectors from the penultimate network layer for each known class
• Fit a Weibull distribution to the tail of distances between correct activations and class mean activation vectors
• Recalibrate scores using Weibull CDF probabilities
• Introduce a synthetic 'unknown' class with a score derived from the complement of known class probabilities This allows the model to explicitly estimate the probability that an input belongs to no known class.

An alternative to EVT-based methods that introduces the concept of reciprocal points—learnable representations for the 'unknown' space that are mutually exclusive with known class prototypes. The training objective pushes known class embeddings toward their prototypes while pulling them away from reciprocal points, creating a margin of separability between known and unknown space. This approach avoids the statistical assumptions of EVT and learns the open space boundary directly through gradient descent, often yielding more robust rejection in high-dimensional embedding spaces.