Risk-Coverage Curve

The curve is generated by applying a confidence threshold to a model's softmax output. Inputs with a maximum predicted probability below the threshold are rejected. As the threshold increases, coverage (the fraction of inputs predicted on) decreases, while the model only handles the 'easy' cases it is highly confident about. This creates a monotonic curve where risk typically decreases as coverage drops.

The AURC is the primary summary metric for selective performance. It integrates the risk over all possible coverage levels, providing a single scalar value. A lower AURC indicates a superior selective classifier.

• Perfect AURC: Zero, achieved by a model that has zero risk at full coverage.
• Random AURC: Equivalent to the average risk of the baseline model.
• Excess AURC (E-AURC): The difference between the model's AURC and the optimal AURC, measuring the room for improvement.

A critical operational point is the maximum coverage achievable while maintaining a pre-defined, acceptable risk level. For a safety-critical application requiring 1% error, the curve directly answers: 'What percentage of cases can we automate?' This allows direct comparison between models: the one with higher coverage at the target risk is superior for that deployment context.

The risk-coverage curve disentangles two distinct model properties:

• Discrimination: The model's raw ability to separate classes, reflected in the curve's overall shape and AURC.
• Confidence Calibration: The alignment of predicted probability with actual accuracy. A miscalibrated but highly discriminative model may have a poor curve because its confidence scores are not reliable for setting a rejection threshold.

Temperature scaling is often applied post-hoc to fix calibration and improve the curve without retraining.

The curve can be plotted with two different y-axes:

• Selective Risk: The error rate on accepted inputs (1 - accuracy). This is the standard view, where the curve slopes downward.
• Selective Accuracy: The accuracy on accepted inputs. This curve slopes upward as coverage decreases.

Both represent the same trade-off, but risk is preferred in safety literature as it directly quantifies the cost of failure.

While the AUROC measures a model's ability to discriminate between classes across all thresholds, the risk-coverage curve evaluates the practical utility of the model's confidence scores for abstention. A model with a high AUROC can still have a poor risk-coverage curve if its confidence estimates are noisy or miscalibrated. The risk-coverage curve is a more direct measure of deployability in cost-sensitive environments.