Openness Measure

The Openness Measure quantifies the proportion of classes the model must reject versus those it must identify. It is formally defined as:

• Formula: Openness = 1 - sqrt((2 × N_train) / (N_test + N_target))
• N_train: Number of known classes used during training
• N_test: Total number of classes appearing during evaluation
• N_target: Number of classes the system is actually required to identify

A score of 0% represents a fully closed-set problem, while scores approaching 100% indicate extreme openness where most classes are unknown.

Without a standardized openness metric, comparing open set recognition algorithms is unreliable. The Openness Measure provides a single scalar value that captures evaluation difficulty:

• Low Openness (< 10%): Few unknowns; favors conservative classifiers
• Medium Openness (10-30%): Balanced known-to-unknown ratio
• High Openness (> 30%): Many unknowns; stresses rejection mechanisms

This prevents researchers from cherry-picking easy splits and enables reproducible science across different datasets and domains.

The Openness Measure directly correlates with Open Space Risk—the probability of misclassifying an unknown sample as a known class. Key dynamics include:

• As openness increases, the volume of unlabeled feature space expands
• Models must balance tight class boundaries against broad rejection coverage
• Higher openness demands more sophisticated Extreme Value Theory (EVT) calibration

The measure helps practitioners select appropriate rejection thresholds before deployment in dynamic environments.

Constructing a valid open set benchmark requires careful adherence to the openness protocol:

• Class Partitioning: Randomly split all available classes into known and unknown subsets
• Openness Calculation: Compute the measure using the formal definition to verify the intended difficulty
• Balanced Sampling: Ensure equal representation of known and unknown samples in the test set
• Threshold Independence: Evaluate using metrics like AUROC that do not depend on a single operating point

This protocol ensures that reported performance reflects genuine open set capability rather than dataset artifacts.

The Openness Measure is domain-agnostic and applies wherever unknown class rejection is critical:

• RF Fingerprinting: Identifying rogue or previously unseen emitters in spectrum monitoring
• Facial Recognition: Rejecting impostors not present in the gallery set
• Autonomous Driving: Detecting novel obstacles absent from training data
• Medical Diagnosis: Flagging rare pathologies as unknowns rather than forcing misclassification

Each domain uses the same openness calculation to define evaluation difficulty, enabling cross-domain algorithm transfer.

While the Openness Measure standardizes difficulty, practitioners must consider its limitations:

• Semantic Similarity Ignored: The measure treats all unknown classes equally, ignoring how visually or semantically similar they are to known classes
• Class Count Sensitivity: The measure depends on the number of classes, not the number of samples per class
• Single Scalar Limitation: A single number cannot capture the full complexity of an open set problem

Complementary metrics like semantic openness and feature space overlap are active areas of research to address these gaps.