AUROC

AUROC evaluates classifier performance across all possible decision thresholds simultaneously, eliminating the need to arbitrarily select a single operating point.

• Plots True Positive Rate (TPR) against False Positive Rate (FPR) as the discrimination threshold varies from 0 to 1
• A single AUROC value summarizes the entire ROC curve, providing a holistic performance measure
• Critical for open set recognition where the optimal rejection threshold for unknown emitters is unknown a priori
• Contrasts with metrics like accuracy or F1-score, which depend on a fixed threshold choice

AUROC is invariant to the prior class distribution, making it robust for evaluating models on highly imbalanced datasets common in emitter identification.

• The metric depends only on the ranking of predictions, not their absolute magnitudes
• A model's AUROC remains identical whether the test set has 1% or 50% unknown emitters
• This property is essential in spectrum surveillance where known signals vastly outnumber unknown threats
• Contrasts with accuracy, which can be misleadingly high on imbalanced data by simply predicting the majority class

AUROC equals the probability that a randomly chosen positive instance (known emitter) receives a higher confidence score than a randomly chosen negative instance (unknown emitter).

• An AUROC of 0.95 means a known emitter is ranked higher than an unknown 95% of the time
• This interpretation directly maps to the open set recognition goal: separating knowns from unknowns
• Provides an intuitive, non-parametric measure of class separability in the learned feature embedding space
• Equivalent to the Wilcoxon-Mann-Whitney U statistic normalized by the product of class counts

AUROC directly quantifies a model's ability to manage open space risk—the probability of labeling an unknown emitter as a known class.

• A higher AUROC indicates tighter decision boundaries around known classes, reducing open space volume
• Models with AUROC below 0.5 are performing worse than random guessing, indicating severe open space risk
• The ROC curve's shape reveals trade-offs: a steep initial rise indicates low FPR at conservative thresholds
• In Weibull-calibrated systems like OpenMax, AUROC validates the quality of the fitted extreme value distributions

AUROC can present an overly optimistic view of classifier performance when the negative class (unknowns) heavily outnumbers the positive class (knowns).

• The False Positive Rate denominator is the total number of negatives, so a small absolute number of false alarms yields a deceptively low FPR
• In open set emitter recognition with millions of background signals, even a 0.1% FPR generates thousands of false alarms
• Precision-Recall AUC is often a more informative complementary metric for extreme class imbalance
• Practitioners should examine the ROC curve's leftmost region (low FPR) rather than relying solely on the summary AUROC value

AUROC serves as a direct proxy for the discriminative quality of learned feature embeddings in deep learning-based open set recognition systems.

• Contrastive learning objectives like SupCon and triplet loss directly optimize for high AUROC by maximizing inter-class separation
• Angular margin losses (ArcFace, CosFace) enforce hyperspherical constraints that improve AUROC by reducing intra-class variance
• The metric validates distance-based rejection logic: high AUROC implies Mahalanobis distance or cosine similarity thresholds cleanly separate knowns from unknowns
• Monitoring AUROC during training provides a reliable signal for early stopping and hyperparameter tuning

An alternative to the ROC curve that plots Precision against Recall at various thresholds. Unlike AUROC, it focuses exclusively on the positive (minority) class and is more informative for highly imbalanced datasets where the number of known transmitters vastly outweighs unknown emitters.

• Precision: TP / (TP + FP)
• Recall: TP / (TP + FN)
• Area Under the Precision-Recall Curve (AUPRC) is the summary metric

The harmonic mean of Precision and Recall, providing a single threshold-dependent metric that balances false positives and false negatives. In open set emitter recognition, the F1 score is often maximized to find the optimal operating point on the ROC curve.

• Formula: 2 * (Precision * Recall) / (Precision + Recall)
• Ranges from 0 to 1, with 1 being perfect
• Best used when both false alarms and missed intrusions carry equal cost

The point on the ROC curve where the False Positive Rate (FPR) equals the False Negative Rate (FNR). EER provides a single-number summary of biometric and authentication system performance.

• Lower EER = Better Performance
• Commonly used in speaker verification and device fingerprinting
• Derived directly from the ROC curve at the intersection with the diagonal line FPR = FNR

A variant of the ROC curve that plots False Rejection Rate (FRR) against False Acceptance Rate (FAR) on a logarithmic scale. DET curves are standard in NIST speaker recognition evaluations and physical-layer authentication benchmarks.

• Emphasizes performance in low-FAR regions critical for security
• A straight line on a normal deviate scale indicates Gaussian score distributions
• Used to set operational thresholds for zero-trust device authentication

A metric designed specifically for open set recognition that measures the correct classification of known classes while simultaneously rejecting unknown classes. It combines top-1 accuracy on knowns with the true negative rate on unknowns.

• OSCR = f(Correct Known, Correct Unknown Rejection)
• Addresses the limitation of AUROC which ignores closed-set accuracy
• Essential for evaluating OpenMax and prototype-based rejection systems

A single statistic that captures the performance of a diagnostic test at a specific threshold: J = Sensitivity + Specificity - 1. It identifies the optimal cut-point on the ROC curve that maximizes the vertical distance from the diagonal chance line.

• Range: 0 (useless) to 1 (perfect)
• Maximizing J finds the threshold with the best combined true positive and true negative rates
• Used to operationalize AUROC into a deployable decision boundary for emitter identification