Open Set Classification Rate

OSCR evaluates a classifier's performance across two simultaneous tasks: correctly identifying known modulation schemes and detecting unknown signal types. Unlike closed-set accuracy, which assumes all test classes were seen during training, OSCR penalizes models that confidently misclassify novel signals as known classes. The metric is typically computed as a function of the rejection threshold, producing a curve that shows the trade-off between Conditional Classification Rate (CCR) on knowns and False Positive Rate (FPR) on unknowns.

OSCR is not a single scalar value but a parametric curve that varies with the novelty detection threshold. As the rejection threshold is tightened:

• Higher rejection rates correctly filter more unknown signals
• Lower CCR may result if known signals are incorrectly rejected

The area under the OSCR curve provides a threshold-independent summary statistic, analogous to AUROC, for comparing open set classifiers.

OSCR directly measures a model's exposure to open space risk—the danger of labeling an unknown input as a known class. A high OSCR indicates that the model maintains a compact decision boundary around known classes while leaving the remainder of the feature space as a rejection zone. This is critical in spectrum monitoring applications where new modulation types, jamming signals, or interference patterns appear regularly.

OSCR is closely related to the Detection Error Tradeoff (DET) curve and Receiver Operating Characteristic (ROC) analysis. While ROC plots True Positive Rate against False Positive Rate for binary novelty detection, OSCR explicitly incorporates multi-class accuracy on knowns into the evaluation. This makes OSCR more informative than standalone novelty detection metrics when the system must both classify and reject.

The reliability of OSCR as an evaluation metric depends heavily on confidence calibration. A poorly calibrated model may produce overconfident SoftMax scores for unknown inputs, degrading OSCR performance even if the underlying feature representations are separable. Techniques like temperature scaling and entropic open-set loss directly improve OSCR by ensuring that low confidence reliably indicates novelty.

OSCR enables fair comparison between different open set recognition approaches:

• OpenMax with Weibull-tail modeling
• Reciprocal Point Learning with distance-based rejection
• Objectosphere loss with feature magnitude separation
• Deep Ensembles with predictive variance

Each method can be evaluated on the same OSCR curve, revealing performance differences at various operating points relevant to deployment requirements.

The fundamental classification paradigm where a model must accurately identify known classes while simultaneously detecting and rejecting samples from unknown classes not seen during training. Unlike closed-set systems that force a prediction, open set recognition introduces a rejection option. The Open Set Classification Rate directly measures performance in this paradigm by penalizing both misclassifications of known signals and the erroneous acceptance of novel modulation schemes.

A deep learning layer that replaces the standard SoftMax function by recalibrating activation vectors using Extreme Value Theory. It fits a Weibull distribution to the distance of correct classifications from their class mean, enabling the model to estimate the probability that an input belongs to an unknown class. The Open Set Classification Rate captures how well this recalibrated probability threshold separates known modulations from novel signal types.

The task of identifying input samples that differ significantly from the training data distribution. Techniques include:

• ODIN: Applies temperature scaling and small adversarial perturbations to amplify SoftMax score differences
• Energy-Based Models: Use Helmholtz free energy as a discriminative score
• Mahalanobis Distance: Accounts for class covariance structure for statistically informed detection The Open Set Classification Rate provides a unified metric for comparing these detection strategies.

The model's uncertainty arising from a lack of knowledge or data, which is reducible with more training samples. This contrasts with aleatoric uncertainty from inherent noise. In open set classification, high epistemic uncertainty signals an unknown modulation type. Methods for quantification include:

• Deep Ensembles: Variance across multiple randomly initialized networks
• Evidence Deep Learning: Dirichlet distributions over class probabilities The Open Set Classification Rate implicitly evaluates uncertainty calibration quality.

The theoretical risk of labeling an unknown input as a known class, quantified as the relative measure of the feature space far from any known training data that is nonetheless classified as known. Minimizing open space risk is the core objective of open set classifier design. The Open Set Classification Rate operationalizes this theoretical concept into a practical, single-scalar metric by jointly measuring known-class accuracy and the false acceptance rate of unknown signals.

The technique of aligning a model's predicted probability with the actual likelihood of correctness. Key methods include:

• Temperature Scaling: Divides logits by a learned scalar to soften SoftMax probabilities
• Entropic Open-Set Loss: Forces high-entropy uniform distributions for unknown samples Proper calibration ensures that a low confidence score reliably indicates an unknown input, making the rejection threshold in the Open Set Classification Rate meaningful and robust across varying signal-to-noise conditions.