Isotonic Regression

This is the canonical use case. Isotonic regression is applied to the raw scores (logits) of a pre-trained classifier to produce calibrated probabilities. It fits a piecewise constant, non-decreasing function that maps the classifier's outputs to probabilities that accurately reflect the true likelihood of an event.

• Process: A held-out calibration set is used to fit the isotonic model. For each input, the classifier's score and the true binary label form the training data for the regression.
• Advantage over parametric methods: Unlike Platt scaling (logistic regression), it makes no assumption about the sigmoidal shape of the miscalibration, allowing it to correct complex, non-linear miscalibration patterns.

Isotonic regression is used to calibrate risk prediction models where a monotonic relationship between a score and risk probability is clinically required. For example, converting a composite health score into a well-calibrated probability of disease onset or mortality.

• Example: A model outputs a "severity score" from 1 to 100 for a patient. Isotonic regression maps these scores to calibrated probabilities of ICU admission, ensuring that a score of 80 always implies a higher risk than a score of 60.
• Benefit: Provides clinicians with reliable, interpretable probabilities for decision-making, adhering to the natural monotonic constraint that higher scores mean higher risk.

In financial risk modeling, credit scores must map monotonically to the probability of default (PD). Isotonic regression is used to transform the outputs of complex machine learning models into PDs that satisfy this regulatory and business requirement.

• Regulatory Compliance: Regulations like Basel II/III require calibrated PD estimates. Isotonic regression ensures the monotonicity demanded by credit risk frameworks.
• Process: A model predicts a default propensity score. Isotonic regression on historical default data produces the final PD, guaranteeing that a customer with a higher propensity score receives a higher PD.

Ranking models for ads or recommendations often output scores that need to be converted into accurate estimated CTRs for bid optimization and inventory pricing. Isotonic regression calibrates these scores using historical impression/click data.

• Use Case: A deep learning model scores an ad's relevance. The raw score is not a probability. Isotonic regression calibrates it to a true CTR (e.g., 0.05 means a 5% click probability).
• Business Impact: Enables accurate value estimation for real-time bidding systems, where bids are often calculated as bid = CTR * value_per_click.

Isotonic regression can be integrated with conformal prediction to create more efficient (tighter) prediction sets. While conformal prediction provides coverage guarantees, it doesn't optimize interval size. Calibrating the underlying model's scores with isotonic regression often leads to scores that better reflect uncertainty, resulting in smaller, more precise prediction sets while maintaining the same coverage guarantee.

• Mechanism: Better-calibrated probabilities allow the conformal score function (e.g., 1 - p_true) to more accurately rank examples by uncertainty.
• Result: For a target 90% coverage, the prediction sets generated from a calibrated model will typically contain fewer irrelevant labels compared to an uncalibrated model.

Isotonic regression is applied to calibrate confidence scores for LLM generations, such as the probability assigned to a multiple-choice answer or the confidence in a factual statement. This is critical for trustworthy AI and enabling models to express meaningful uncertainty.

• Challenge: LLMs are notoriously overconfident. Their softmax probabilities do not reflect true likelihoods.
• Application: On a validation set of questions, the model's predicted probability for the chosen answer is recorded along with correctness (1/0). Isotonic regression fits a mapping from these flawed probabilities to calibrated ones.
• Outcome: A calibrated LLM can more reliably use its own confidence score to decide when to abstain or seek human help, a key component of selective calibration.

Platt scaling (or sigmoid calibration) is a parametric post-hoc calibration method for binary classifiers. It fits a logistic regression model to the classifier's raw output scores (logits) to map them to calibrated probabilities. Unlike isotonic regression, it assumes a specific sigmoidal shape for the calibration function.

• Parametric vs. Non-Parametric: Assumes a specific functional form (sigmoid), whereas isotonic regression is non-parametric.
• Application: Primarily used for binary classification. Extension to multi-class settings (e.g., OvR - One-vs-Rest) is common.
• Efficiency: Requires fitting only two parameters (slope and intercept), making it less flexible but more data-efficient than isotonic regression on very small calibration sets.

Temperature scaling is a single-parameter, post-hoc calibration technique that applies a scalar 'temperature' (T) to the logits of a neural network before the softmax function. A T > 1 softens the output distribution (reducing confidence), while T < 1 sharpens it.

• Simplicity: A lightweight, widely used method for modern neural networks, especially in multi-class settings.
• Limitation: Only adjusts the 'spread' of the distribution, maintaining the original ranking of classes. It cannot correct for systematic biases in the original probabilities.
• Relation to Isotonic: Isotonic regression is more flexible and can learn non-linear, non-monotonic mappings (though constrained to be non-decreasing), whereas temperature scaling is a global linear transformation.

Expected Calibration Error (ECE) is a primary scalar metric for quantifying miscalibration. It bins predictions based on their confidence score and computes the weighted average of the absolute difference between average confidence and empirical accuracy within each bin.

• Calculation: ECE = Σ (n_b / N) * |acc(b) - conf(b)|, where n_b is the number of samples in bin b, N is total samples, acc(b) is the accuracy in bin b, and conf(b) is the average confidence in bin b.
• Diagnostic Use: A well-calibrated model has an ECE near zero. It is the standard benchmark for evaluating methods like isotonic regression.
• Limitation: Sensitive to the number and placement of bins. Adaptive binning schemes can mitigate this.

A reliability diagram is the primary visual tool for diagnosing calibration. It plots a model's average predicted confidence (x-axis) against its observed empirical accuracy (y-axis) across multiple confidence bins.

• Interpretation: Points on the diagonal (y=x) indicate perfect calibration. Deviations below the diagonal signify overconfidence; deviations above signify underconfidence.
• Visualizing Isotonic Fit: The output of isotonic regression is a piecewise constant function that can be overlaid on a reliability diagram, showing how it 'pushes' the original miscalibrated curve toward the ideal diagonal.
• Foundation: The graphical counterpart to the numerical ECE metric.

Post-hoc calibration is the overarching paradigm for applying calibration transformations after a model is trained, without modifying its internal parameters. Isotonic regression is a canonical non-parametric method within this family.

• Key Principle: Decouples the task of achieving high accuracy (discrimination) from the task of achieving calibrated probabilities (calibration).
• Workflow: Requires a held-out calibration set, distinct from training and test data, to fit the calibration mapping (e.g., the isotonic function).
• Other Methods: Includes Platt scaling, temperature scaling, and beta calibration. The choice depends on data size, model type, and the nature of miscalibration.

A calibration set (or hold-out calibration set) is a dedicated dataset used exclusively to fit the parameters of a post-hoc calibration method. It is a critical component for techniques like isotonic regression and Platt scaling.

• Data Partitioning: Typically drawn from the same distribution as the training data but held out from both the training and test/validation sets.
• Size Requirements: Non-parametric methods like isotonic regression generally require more calibration data (hundreds to thousands of samples) than parametric methods like temperature scaling to avoid overfitting.
• Operational Role: In production, maintaining a representative calibration set is essential for periodic recalibration to combat calibration drift.