Open-Set Recognition

Traditional classifiers operate under the closed-world assumption, where all test classes are present in training. OSR rejects this, acknowledging that in real-world RF environments, rogue or previously unseen emitters will inevitably appear. A closed-set model forced to classify an unknown device will confidently misclassify it as a known emitter, creating a critical security blind spot. OSR architectures explicitly model the space of 'unknown' inputs.

The core mathematical principle of OSR is balancing empirical risk (error on known classes) against open space risk (labeling unknown space as known). A classifier must define a bounded region for each known emitter class. The open space risk is the relative measure of the feature space labeled as 'known' compared to the infinite space of 'unknown'. Effective OSR minimizes this risk by creating compact, tight decision boundaries around known fingerprint clusters.

OSR systems rely on a rejection layer that evaluates the model's confidence before assigning a label. Common techniques include:

• Softmax Thresholding: Rejecting predictions where the maximum posterior probability falls below a calibrated threshold.
• OpenMax Layer: Replacing the standard SoftMax with a layer that estimates the probability of an input belonging to an 'unknown' class by analyzing the distribution of activation vectors in the penultimate layer.
• Distance-Based Rejection: Rejecting samples whose embedding distance to the nearest class prototype exceeds a learned radius.

Rather than learning class boundaries directly, metric learning approaches train a neural network to map RF fingerprints into an embedding space where distance corresponds to device similarity. Known emitters form tight, distinct clusters. Unknown devices are detected because their embeddings fall in low-density regions far from any known cluster centroid. Triplet loss and prototypical networks are foundational architectures that naturally support open-set rejection by establishing a distance metric for novelty.

To statistically model the boundary between known and unknown, OSR systems often apply Extreme Value Theory. EVT fits a Weibull distribution to the tail of the distance scores between correct classifications and their class centroids. This allows the model to compute a calibrated probability that a new sample belongs to an unknown class, rather than using an arbitrary threshold. The OpenMax algorithm is the canonical example of EVT-based open-set recognition.

While related, these are distinct tasks. Novelty detection is a one-class problem: detecting any deviation from a single 'normal' class. Open-Set Recognition is a multi-class problem: accurately classifying among K known classes while simultaneously detecting the (K+1)th unknown class. In RF fingerprinting, OSR is the operational requirement—the system must say 'This is Device A' or 'This is an unknown rogue,' not merely 'This is anomalous.'

The security task of identifying a rogue device attempting to impersonate a legitimate transmitter by copying its higher-layer credentials. Clone detection is thwarted by verifying the immutable physical-layer fingerprint. Open-set recognition provides the formal framework for this: a cloned device's hardware impairments will never perfectly match the original, creating an unknown class that must be detected and rejected rather than silently accepted as the legitimate emitter.

A metric learning technique that trains a neural network to map RF fingerprints into a high-dimensional space where signals from the same device are clustered together and signals from different devices are pushed apart by a defined margin. This embedding space is ideal for open-set recognition: unknown emitters naturally fall into low-density regions of the space, allowing a distance-based novelty detector to reject them without requiring a separate background class during training.