Selective Classification

Conformal prediction provides a distribution-free framework for constructing prediction sets with finite-sample coverage guarantees. Instead of a single label, the model outputs a set of plausible classes, and abstention occurs when the set is empty or exceeds a size constraint.

• Split Conformal: Uses a held-out calibration set to determine thresholds
• Adaptive Prediction Sets (APS): Adjusts set sizes based on instance difficulty
• Mondrian Conformal: Provides conditional coverage across subpopulations

The prediction set size itself becomes a natural abstention signal—large sets indicate high epistemic uncertainty, which correlates strongly with non-membership in the training data.

Standard models are trained to minimize error on all inputs, but selective classifiers benefit from training objectives that explicitly model abstention. This includes:

• SelectiveNet: A deep architecture with an integrated rejection head trained end-to-end to optimize the risk-coverage tradeoff
• Deep Gamblers: Models that learn to reserve a budget of abstentions for the most uncertain inputs
• Auxiliary Rejection Loss: Penalizes confident wrong answers more heavily than abstentions

These training paradigms produce models that are inherently calibrated for abstention, rather than retrofitting thresholds onto a standard classifier, resulting in tighter privacy-utility operating points.

A distribution-free framework that produces prediction sets with guaranteed coverage at a user-specified confidence level. Unlike softmax thresholding, conformal prediction provides rigorous finite-sample validity—the true label falls within the prediction set with probability at least 1-α. When the prediction set contains multiple classes or is empty, the model abstains, directly masking the overconfident signals exploited by membership inference attacks. The framework uses a calibration set held out from training to compute nonconformity scores, enabling abstention decisions with formal statistical backing rather than heuristic uncertainty thresholds.

A measure of uncertainty in a model's output probability distribution, calculated as H(p) = -Σ p_i log(p_i). Training samples typically exhibit lower entropy (higher confidence) than non-training samples, creating the primary signal that membership inference attacks exploit. Selective classification uses entropy thresholds to identify and abstain on high-entropy inputs where the model lacks sufficient evidence. Key distinctions include:

• Aleatoric uncertainty: Inherent noise in the data distribution
• Epistemic uncertainty: Model uncertainty due to limited knowledge, reducible with more data Monitoring entropy distributions across training and test sets reveals memorization patterns and guides abstention threshold calibration.

A technique that identifies inputs dissimilar to the training distribution, enabling the model to throttle predictions on anomalous queries. OOD detection methods include:

• Mahalanobis distance in feature space
• Energy-based models that assign lower scores to OOD samples
• Gradient-based detection analyzing backpropagation patterns When combined with selective classification, OOD detection prevents attackers from probing the model with adversarially crafted membership inference queries that lie near decision boundaries. This dual-layer defense ensures that both in-distribution uncertain samples and out-of-distribution probes trigger abstention, reducing the attack surface for privacy leakage.

A post-hoc calibration method that smooths model output probabilities by dividing logits by a learned temperature parameter T. When T > 1, the probability distribution softens, reducing the confidence gap between training and non-training samples that membership inference attacks exploit. Temperature scaling is optimized on a held-out validation set to minimize expected calibration error (ECE). For selective classification, calibrated probabilities provide more reliable confidence estimates for abstention decisions. However, temperature scaling alone does not eliminate membership information—it must be combined with differential privacy or explicit abstention mechanisms for robust protection.

The field of estimating and representing confidence bounds on model predictions, distinguishing between:

• Aleatoric uncertainty: Irreducible noise inherent in the data
• Epistemic uncertainty: Model uncertainty from limited data or capacity, reducible with more training UQ methods include Bayesian neural networks, Monte Carlo dropout, and deep ensembles that produce predictive distributions rather than point estimates. For selective classification, UQ provides the theoretical foundation for abstention decisions—when epistemic uncertainty is high, the model should abstain. This directly counters membership inference by denying attackers the high-confidence signals they require, as memorized training samples typically exhibit lower epistemic uncertainty than genuine out-of-sample inputs.

The alignment between a model's predicted confidence and its empirical accuracy. A perfectly calibrated model that predicts 80% confidence is correct exactly 80% of the time. Modern neural networks are notoriously miscalibrated, often producing overconfident predictions that membership inference attacks exploit. Calibration metrics include:

• Expected Calibration Error (ECE): Weighted average of confidence-accuracy gaps across bins
• Maximum Calibration Error (MCE): Worst-case deviation in any confidence bin
• Brier Score: Mean squared error between predicted probabilities and outcomes Well-calibrated models exhibit smaller confidence discrepancies between training and non-training samples, reducing the signal available for membership inference. Selective classification leverages calibration to set principled abstention thresholds.