Isotonic Regression

The core principle of isotonic regression is the monotonicity constraint: if a model assigns a higher raw score to instance A than to instance B, the calibrated probability for A must be greater than or equal to that for B. This preserves the original ranking of the classifier.

• Preserves AUC: Because the ordering is strictly maintained, the Area Under the ROC Curve (AUC) remains unchanged after calibration.
• No functional form: Unlike Platt Scaling (sigmoid) or Temperature Scaling (single parameter), isotonic regression imposes no parametric shape on the mapping function.
• Learned from data: The exact shape of the non-decreasing function is determined entirely by minimizing a loss function on a held-out calibration set.

The output of isotonic regression is a step function—a series of flat probability plateaus separated by discrete jumps. This is fundamentally different from the smooth sigmoid curves produced by parametric methods.

• Bucket formation: The algorithm groups contiguous raw scores into 'buckets' or 'bins' and assigns a single calibrated probability to each bucket.
• Pool Adjacent Violators (PAVA): The standard algorithm for fitting isotonic regression. It iteratively merges adjacent bins that violate the monotonicity constraint until the sequence is non-decreasing.
• Flat regions: The step function creates flat regions where the model's confidence is constant, which can be a disadvantage if the underlying relationship is smooth.

The Pool Adjacent Violators Algorithm (PAVA) is the canonical method for computing the isotonic regression solution. It is computationally efficient and guarantees a global optimum.

• Initialization: Sort all instances by their raw uncalibrated scores. Each instance starts as its own bin with a target value (0 or 1).
• Violation check: Iterate through the sorted bins. If the current bin's mean target value is less than the previous bin's mean, a violation of monotonicity has occurred.
• Pooling: Merge the violating adjacent bins into a single pool and recompute the pooled mean. Then, check backward against the new predecessor.
• Result: The final sequence of pooled means forms the non-decreasing step function. The algorithm runs in O(n) time after sorting.

The primary weakness of isotonic regression is its tendency to overfit, especially in low-data regions. Because it learns a parameter for nearly every data point, it has extremely high variance.

• Data requirement: Isotonic regression requires a significantly larger calibration set than parametric methods like Platt or Temperature Scaling to produce stable estimates.
• Tail instability: The step function is particularly unreliable at the extreme ends of the score distribution where data is sparse.
• Mitigation: Regularized isotonic regression or using it only when the calibration set is large (e.g., >10,000 samples) can reduce overfitting. Cross-validation is essential for determining the appropriate binning granularity.

Isotonic regression occupies one end of the flexibility spectrum for calibration methods. Choosing the right method depends on the nature of the miscalibration and the size of the calibration set.

• Isotonic Regression: Most flexible, no shape assumption. Best for large datasets with complex, non-sigmoidal miscalibration patterns. Highest risk of overfitting.
• Platt Scaling: Assumes a sigmoid relationship between raw scores and probabilities. More robust with smaller datasets. Fits two parameters (A, B) to a logistic function.
• Temperature Scaling: A single-parameter simplification of Platt Scaling (no bias term). Only adjusts the sharpness of the softmax distribution. Extremely robust, cannot change the rank order or the shape of the confidence curve.

Isotonic regression naturally extends to multiclass calibration through a one-vs-all or pairwise coupling strategy, though it is more complex than the binary case.

• One-vs-All: Train a separate isotonic regressor for each class, treating it as a binary problem (class k vs. not class k). The resulting K probability vectors must then be normalized (e.g., via softmax) to sum to 1.
• Pairwise Coupling: Fit binary isotonic models for every pair of classes (i, j) to estimate P(i | i or j). These pairwise probabilities are then combined into a full multiclass distribution.
• Classwise calibration: Isotonic regression can achieve classwise calibration, ensuring that the predicted probability for each individual class matches its empirical frequency.

The primary metric for evaluating calibration quality. ECE partitions predictions into M equal-width bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin.

• Lower ECE indicates better calibration
• Sensitive to binning strategy (number of bins)
• Does not distinguish between overconfidence and underconfidence
• A perfectly calibrated model achieves ECE = 0

A visual diagnostic tool that plots predicted probability on the x-axis against observed frequency on the y-axis. A perfectly calibrated model follows the identity diagonal (y = x).

• Bars above the diagonal indicate underconfidence
• Bars below the diagonal indicate overconfidence
• Often paired with a histogram showing sample density per bin
• Provides immediate visual intuition that ECE summarizes numerically

A parametric alternative to isotonic regression that fits a logistic regression model to the raw classifier scores. Unlike isotonic regression, Platt scaling assumes a sigmoid-shaped mapping.

• Works well when the distortion is mild and sigmoidal
• Requires fewer data points than isotonic regression
• Less prone to overfitting on small calibration sets
• Fails when the true miscalibration pattern is non-sigmoidal

The simplest post-hoc calibration method for neural networks. A single scalar parameter T > 0 divides all logits before the softmax, sharpening (T < 1) or softening (T > 1) the distribution.

• Does not change the ranking of predictions (accuracy unchanged)
• Optimized on a held-out validation set via NLL minimization
• An extension of Platt scaling to multiclass problems
• Cannot correct arbitrary miscalibration shapes

A class of probabilistic predictors that combines isotonic regression with Venn prediction to produce multiprobability outputs with proven calibration guarantees under the i.i.d. assumption.

• Outputs an interval of probabilities rather than a point estimate
• Guarantees validity without distributional assumptions
• Built directly on top of isotonic regression as the underlying calibrator
• Useful in high-stakes applications requiring formal guarantees

A strictly proper scoring rule that measures the mean squared difference between predicted probabilities and actual binary outcomes. Unlike ECE, the Brier score jointly evaluates both calibration and refinement (discrimination).

• Range: 0 (perfect) to 1 (worst)
• Decomposable into calibration loss + refinement loss
• Penalizes confident errors more heavily than hesitant ones
• A lower Brier score always indicates a better probabilistic forecast