Precision-Recall AUC

The ROC AUC measures a model's ability to distinguish between classes across all thresholds, but it can be deceptively optimistic on imbalanced data because it rewards correct predictions of the abundant majority class. Precision-Recall AUC focuses exclusively on the minority class performance, ignoring true negatives entirely. When the positive class is rare (<1%), a high ROC AUC often masks a model with terrible precision, making PR AUC the preferred metric for fraud detection.

The F1 Score is the harmonic mean of precision and recall, providing a single-point metric at a specific decision threshold. While PR AUC summarizes performance across all thresholds, the F1 score is used to select the optimal operating point for production. A model with a high PR AUC may still have a poor F1 score if the chosen threshold is poorly calibrated. Techniques like threshold moving are used to maximize F1 on a validation set after the PR AUC has validated the overall model quality.

Average Precision is mathematically equivalent to the area under the Precision-Recall curve. It summarizes the precision-recall trade-off as a single scalar value, calculated as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight. This metric is the default ranking loss for many object detection and information retrieval tasks and is more sensitive to the ordering of positive predictions than ROC AUC.

A critical property of the PR AUC is that its baseline is not fixed at 0.5 like the ROC AUC. The no-skill value for PR AUC is equal to the positive class prevalence. For a dataset with 1% fraud, a random classifier achieves a PR AUC of 0.01. This dynamic baseline makes PR AUC a more honest metric: a score of 0.10 on a 1% positive dataset represents a 10x improvement over random, whereas a 0.50 ROC AUC would be completely useless.

The Precision-Recall curve is constructed by sorting predictions by descending confidence score and calculating precision and recall at each unique score threshold. Because precision does not monotonically decrease with recall, the raw curve is often jagged. Interpolation smooths the curve by taking the maximum precision for any recall level greater than the current one, ensuring a monotonically decreasing function for cleaner visualization and area calculation.

PR AUC is inherently aligned with cost-sensitive learning because it penalizes false positives heavily when recall is high. In fraud, the cost of a false positive (blocking a legitimate transaction) is often much lower than a false negative (missing fraud), but both matter. By examining specific regions of the PR curve—such as high-recall areas—teams can evaluate model performance under specific business cost ratios without needing explicit cost matrices during training.