Open Set Recognition

Traditional classifiers operate under a closed-world assumption where all test classes are present during training. OSR rejects this, acknowledging that real-world deployments encounter unknown unknowns.

• Closed Set: 100% of test classes seen in training; model forced to map every input to a known label
• Open Set: Incomplete training knowledge; model must recognize when it encounters something unfamiliar
• Failure Mode: Closed-set models assign high-confidence wrong labels to unknown inputs, creating silent failures
• Practical Impact: An autonomous vehicle trained on common road objects must reject a fallen tree or overturned truck rather than misclassifying it as a known benign object

OSR formalizes the classification problem into three distinct regions of the feature space, each requiring different model behavior.

• Known Knowns (KK): Labeled classes from training data; model must classify with high accuracy
• Known Unknowns (KU): Deliberately identified negative samples or outlier data used during training to shape the rejection boundary
• Unknown Unknowns (UU): Entirely novel classes never seen during training or validation; the model must detect these as unfamiliar and reject them
• The Core Challenge: The UU space is infinite and unbounded, making it impossible to sample exhaustively during training

Open Space Risk quantifies the probability of labeling an unknown input as a known class. Minimizing this risk is the central optimization objective of OSR.

• Definition: The relative measure of positively labeled open space compared to the overall measure of positively labeled space
• Boundedness Principle: To minimize open space risk, the model must bound the recognition region for each known class as tightly as possible
• Extreme Value Theory (EVT) is often applied to model the tail behavior of class distributions, enabling statistical calibration of rejection thresholds
• Weibull Distribution Fitting: Techniques like OpenMax fit Weibull distributions to the distance of correct and incorrect activations per class, recalibrating the softmax layer to include an explicit unknown probability

OpenMax replaces the standard softmax layer with an activation vector that explicitly models the probability of an unknown class, serving as the foundational OSR baseline.

• Meta-Recognition Calibration: For each known class, OpenMax fits a Weibull distribution to the tail of distances between correct and incorrect activation vectors
• Activation Re-weighting: Top activations are re-weighted based on their Weibull CDF probability; the remaining mass is assigned to an explicit unknown class
• Rejection Logic: If the unknown class probability is the highest, or if the top known class probability falls below a threshold, the input is rejected
• Limitation: OpenMax assumes the feature extractor produces meaningful activation magnitudes, which may not hold for all architectures

OSR methods broadly divide into discriminative and generative paradigms, each with distinct trade-offs for bounding open space risk.

• Discriminative Methods: Focus on tightening decision boundaries around known classes using techniques like prototype learning, angular margin penalties, and reciprocal point learning
• Generative Methods: Model the full probability density of known classes, using likelihood thresholds to reject low-density samples; includes autoregressive models, normalizing flows, and diffusion models
• Counterintuitive Finding: Generative models can assign higher likelihood to OOD samples than in-distribution ones, a phenomenon known as the likelihood trap
• Hybrid Approaches: Combine discriminative boundaries with generative density estimation to leverage the strengths of both paradigms

Standard accuracy is insufficient for OSR evaluation. Specialized metrics capture the trade-off between known-class accuracy and unknown-class rejection.

• Open Set Classification Rate (OSCR): Plots correct classification rate against false positive rate as the rejection threshold varies, providing a holistic performance curve
• AUROC for OSR: Area under the receiver operating characteristic for the binary task of distinguishing known from unknown samples
• F1-Measure at Known Rejection Rates: Evaluates classification quality on accepted samples at specific rejection thresholds
• Closed-Set Accuracy vs. Open-Set Rejection: A fundamental trade-off; aggressive rejection improves safety but may degrade throughput on known classes