Calibration of LLMs

Post-hoc calibration applies a transformation to a trained model's outputs without retraining its core parameters. It uses a held-out calibration set to fit simple functions that map raw logits to better-calibrated probabilities.

• Temperature Scaling: Applies a single scalar 'temperature' to soften or sharpen the softmax distribution. It's the most common method for LLMs due to its simplicity and effectiveness.
• Platt Scaling (Sigmoid Calibration): Fits a logistic regression model to the logits, ideal for binary classification tasks.
• Isotonic Regression: Fits a non-parametric, piecewise constant function, powerful for complex miscalibration patterns but prone to overfitting on small datasets.

These methods incorporate calibration objectives directly into the training loss function, aiming to produce intrinsically well-calibrated models.

• Label Smoothing: Replaces hard one-hot labels with a weighted mixture of the true label and a uniform distribution, penalizing overconfidence and often improving calibration.
• Focal Loss: Down-weights the loss for well-classified examples, indirectly mitigating overconfidence, especially in class-imbalanced scenarios.
• Bayesian Neural Networks: Model uncertainty in weights inherently, often leading to better-calibrated predictive uncertainty, though at high computational cost.

Conformal prediction is a distribution-free framework that provides rigorous, statistically valid uncertainty quantification. It generates prediction sets (e.g., multiple possible answers) guaranteed to contain the true label with a user-specified probability (e.g., 90%).

• Unlike scaling methods that adjust a single probability, it outputs a set of plausible labels.
• Provides coverage guarantees that hold under minimal assumptions, making it valuable for high-stakes applications.
• Requires a separate calibration set to compute non-conformity scores.

Combining predictions from multiple models (ensembles) improves accuracy but does not guarantee calibration. The ensemble's averaged probabilities often remain overconfident.

• Post-hoc calibration on ensemble logits: Apply temperature scaling or Platt scaling to the averaged logits of the ensemble members.
• Bayesian Model Averaging: A principled framework that marginalizes over model parameters, typically yielding well-calibrated uncertainty estimates.
• Ensembles are particularly effective for out-of-distribution calibration, as diversity in member models can better capture epistemic uncertainty.

Also known as rejection or selective classification, this approach allows a model to abstain from making a prediction when its confidence is below a threshold. The goal is to maintain high accuracy and calibration only on the subset of instances where it chooses to predict.

• A coverage-calibration trade-off exists: higher confidence thresholds lead to better accuracy on predicted instances but lower overall coverage.
• Critical for deploying LLMs in safety-sensitive domains where incorrect but confident outputs are unacceptable.
• Requires defining a confidence metric (e.g., max softmax probability) and setting an operational threshold.

Calibration is not a one-time fix. Calibration drift occurs when the data distribution shifts in production, degrading calibration performance.

• Continuous Monitoring: Track calibration metrics like Expected Calibration Error (ECE) or Brier Score on a held-out validation stream or via production canaries.
• Automated Recalibration Pipelines: Trigger retraining of the post-hoc calibrator (e.g., refitting the temperature parameter) using recent data when drift is detected.
• Conceptual Framework: This operational practice falls under Calibration in Production, requiring MLOps infrastructure for model and calibrator versioning, data logging, and pipeline orchestration.

Expected Calibration Error (ECE) is the primary scalar metric for quantifying miscalibration. It works by:

• Binning predictions based on their predicted confidence score (e.g., 0.9-1.0).
• For each bin, calculating the absolute difference between the average predicted confidence and the actual empirical accuracy.
• Computing a weighted average of these differences across all bins. A lower ECE indicates better calibration. It is a critical benchmark for comparing calibration techniques.

Post-hoc calibration refers to techniques applied to a trained model's outputs without retraining the model itself. It is the most common approach for LLMs. Key methods include:

• Temperature Scaling: Applies a single scalar to soften or sharpen logits.
• Platt Scaling: Fits a logistic regression model to the outputs.
• Isotonic Regression: Fits a non-parametric, piecewise constant function. These methods require a separate calibration set to learn the correction mapping.

A reliability diagram is the fundamental visual diagnostic tool for calibration. It is a plot where:

• The x-axis represents the model's average predicted confidence within a bin.
• The y-axis represents the corresponding observed empirical accuracy. A perfectly calibrated model's plot follows the 45-degree diagonal. Deviations show the nature of miscalibration:
• Overconfidence: Points below the diagonal (confidence > accuracy).
• Underconfidence: Points above the diagonal (accuracy > confidence).

Proper scoring rules are loss functions that measure the quality of probabilistic forecasts and incentivize the forecaster to report their true beliefs. They are essential for both training and evaluating calibrated models. The two most important are:

• Negative Log-Likelihood (NLL): Penalizes low probability assigned to the correct outcome. It is sensitive to calibration and is often the training objective.
• Brier Score: The mean squared error between predicted probabilities and true binary outcomes. It decomposes into calibration loss and refinement loss.

Conformal prediction is a distribution-free framework that provides rigorous, statistical uncertainty quantification. Instead of producing a single probability, it generates a prediction set guaranteed to contain the true label with a user-specified probability (e.g., 90%). For LLMs, this can be applied to:

• Multiple-choice QA, creating sets of plausible answers.
• Text generation, though more complex. It uses a calibration set to determine the threshold for set inclusion, offering a robust alternative to standard probabilistic calibration.

Calibration in production refers to the operational lifecycle required to maintain calibration after deployment. Key challenges include:

• Calibration Drift: Model confidence becomes miscalibrated due to changing data distributions (dataset shift).
• Monitoring: Continuously tracking metrics like ECE on live traffic.
• Recalibration: Implementing automated calibration pipelines to periodically refit calibration mappings (e.g., temperature) on fresh data. This is a core component of MLOps for reliable, trustworthy AI systems.