Receiver Operating Characteristic (ROC) Curve

The ROC curve is a two-dimensional graph plotting the True Positive Rate (TPR) on the y-axis against the False Positive Rate (FPR) on the x-axis.

• TPR (Sensitivity/Recall): The proportion of actual positives correctly identified. Calculated as TP / (TP + FN).
• FPR (Fall-out): The proportion of actual negatives incorrectly classified as positive. Calculated as FP / (FP + TN). Each point on the curve represents a specific (FPR, TPR) pair corresponding to a distinct decision threshold applied to the classifier's output score.

The ROC curve is generated by sweeping the classifier's discrimination threshold from its minimum to its maximum value.

• At one extreme (lowest threshold), all samples are classified as positive, placing the operating point at the top-right corner (TPR=1, FPR=1).
• At the other extreme (highest threshold), no samples are classified as positive, placing the point at the bottom-left origin (TPR=0, FPR=0).
• The curve connecting these points reveals the trade-off: increasing sensitivity inevitably increases the false alarm rate.

The Area Under the ROC Curve (AUC) is a single scalar metric summarizing the classifier's overall ability to discriminate between classes.

• AUC = 1.0: Represents a perfect classifier that achieves 100% sensitivity with 0% FPR.
• AUC = 0.5: Represents a random classifier with no discriminative power, indicated by the diagonal line y = x.
• AUC < 0.5: Indicates a classifier that performs worse than random chance, suggesting the model has learned an inverted relationship. The AUC is equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one.

The ROC space provides clear visual benchmarks for evaluating classifier quality.

• The Diagonal Baseline: The line connecting (0,0) to (1,1) represents the performance of a random guess. Any useful classifier must operate strictly above this line.
• The Ideal Point: The top-left corner (0,1) is the point of perfect classification.
• Dominance: A classifier's ROC curve is said to dominate another if it is consistently closer to the top-left corner, indicating higher TPR for any given FPR. This allows for direct visual comparison of multiple models without selecting a specific operating point.

A critical property of the ROC curve is its invariance to class prior probabilities. Because both axes are rate-based metrics calculated independently for the positive and negative columns of a confusion matrix, the curve's shape does not change if the proportion of positive to negative instances in the test set is altered. This makes ROC analysis particularly valuable in domains like anomaly detection or rare signal identification, where the target class is severely underrepresented. In contrast, a Precision-Recall curve would shift dramatically with such an imbalance.

While the ROC curve describes global performance, a specific application requires selecting a single operating point. The optimal point is not simply the one closest to (0,1). It is determined by the specific costs of errors.

• Youden's J Index: A common method that maximizes J = Sensitivity + Specificity - 1, which geometrically corresponds to the point on the curve with the maximum vertical distance from the diagonal baseline.
• Cost-Based Selection: In contexts like the Neyman-Pearson Criterion, an operating point is chosen by moving vertically from a maximum tolerable FPR on the x-axis to the curve, thereby maximizing TPR under a strict false alarm constraint.

A scalar aggregate measure of classifier performance across all possible decision thresholds. The AUC represents the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative one.

• AUC = 1.0: Perfect discrimination between classes.
• AUC = 0.5: Performance equivalent to random guessing (the diagonal line).
• AUC < 0.5: The classifier is performing worse than random, indicating inverted predictions.

In likelihood-based classifiers, AUC provides a threshold-agnostic metric to compare the GLRT against the ALRT without committing to a specific operating point.

A tabular layout that explicitly counts the four fundamental outcomes of binary classification: True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN). While the ROC curve visualizes the trade-off between the True Positive Rate (TPR) and False Positive Rate (FPR), the confusion matrix provides the raw counts from which these rates are derived.

For an M-ary modulation classifier, the matrix expands to an M×M grid, revealing specific pairwise confusions—such as misclassifying 16-QAM as 64-QAM—that a single ROC curve cannot capture.

An alternative diagnostic plot that graphs Precision (Positive Predictive Value) against Recall (True Positive Rate). The PR curve is the preferred evaluation tool when dealing with highly imbalanced datasets, a common scenario in spectrum monitoring where a specific modulation type is rare.

• Precision answers: 'Of all signals classified as BPSK, how many were actually BPSK?'
• Recall answers: 'Of all actual BPSK signals, how many did we find?'

Unlike the ROC curve, a random classifier's PR curve baseline is not fixed at 0.5 but depends on the class prevalence.

The decision-theoretic foundation upon which the ROC curve is built. This criterion defines the optimal test as one that maximizes the probability of detection (TPR) while constraining the probability of false alarm (FPR) to a user-defined significance level, α.

In the context of likelihood-based modulation classifiers, the Neyman-Pearson lemma proves that the Likelihood Ratio Test (LRT) is the uniformly most powerful test. The ROC curve is generated by sweeping this constraint α across its full range [0, 1], tracing the resulting (FPR, TPR) pairs.

A practical methodology for selecting a single decision threshold from the ROC curve by assigning explicit misclassification costs. The optimal operating point minimizes the total expected cost:

Total Cost = C_FP × FPR × P(Negative) + C_FN × (1 - TPR) × P(Positive)

• C_FP: Cost of a false alarm (e.g., wasting compute on a decoy signal).
• C_FN: Cost of a missed detection (e.g., failing to identify a hostile radar).

An iso-performance tangent line with slope determined by the cost ratio and class priors identifies the point on the convex hull of the ROC curve that minimizes this cost.

A variant of the ROC curve that plots the False Rejection Rate (FRR) against the False Acceptance Rate (FAR) on a normal deviate scale. This warped coordinate system linearizes Gaussian distributions, making it easier to compare classifier performance at extreme operating points.

DET curves are particularly useful for RF fingerprinting and authentication systems where the acceptable FAR is extremely low (e.g., 10⁻⁶), and the standard ROC curve's linear axes obscure critical performance differences in this high-security region.