Calibration Error

The primary metric for measuring miscalibration. ECE partitions predictions into M confidence bins (e.g., 0-10%, 10-20%) and computes a weighted average of the absolute difference between accuracy and confidence within each bin.

• Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|
• Perfect calibration: ECE = 0
• Overconfidence: confidence > accuracy (positive gap)
• Underconfidence: confidence < accuracy (negative gap)

ECE provides a scalar summary of reliability, but it is sensitive to binning strategy and does not capture within-bin variation.

A visual diagnostic tool that plots observed accuracy against predicted confidence. In a perfectly calibrated model, the plot follows the identity line (y = x).

• Above the diagonal: Model is underconfident (accuracy exceeds confidence)
• Below the diagonal: Model is overconfident (confidence exceeds accuracy)
• Gap size: Indicates severity of miscalibration at that confidence level

Reliability diagrams reveal per-bin miscalibration patterns that a single ECE scalar might obscure, such as severe overconfidence only in high-confidence predictions.

A post-hoc calibration method that applies a single scalar parameter T (temperature) to the model's logits before the softmax function. It does not alter the model's accuracy or ranking of predictions—only the confidence scores.

• T > 1: Softens probabilities, reducing overconfidence
• T < 1: Sharpens probabilities, reducing underconfidence
• Optimization: T is learned on a held-out validation set by minimizing Negative Log Likelihood (NLL)

Temperature scaling is the simplest and most widely effective calibration technique for modern neural networks, requiring no architectural changes.

A parametric calibration method that fits a logistic regression model on top of the raw classifier scores. It transforms the output scores into calibrated probabilities using learned parameters A and B.

• Formula: P(y=1|x) = 1 / (1 + exp(A * f(x) + B))
• Training: Parameters are optimized on a held-out calibration set using maximum likelihood
• Use case: Effective for binary classification and works well with Support Vector Machines (SVMs)

Platt scaling is more flexible than temperature scaling but can overfit on small calibration sets. It is less commonly used for deep neural networks compared to temperature scaling.

A non-parametric calibration method that learns a monotonically increasing piecewise constant function mapping raw scores to calibrated probabilities. It makes no assumptions about the functional form of the miscalibration.

• Constraint: The mapping function must be non-decreasing
• Optimization: Minimizes squared error subject to the monotonicity constraint
• Advantage: Can correct any monotonic distortion in the predicted probabilities
• Risk: Prone to overfitting on small datasets due to its high flexibility

Isotonic regression is preferred when the calibration error is complex and non-sigmoidal, but it requires a larger calibration set than parametric methods.

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and actual binary outcomes. It jointly evaluates discrimination and calibration.

• Formula: BS = (1/N) * Σ (p_i - o_i)², where p_i is predicted probability and o_i is the actual outcome (0 or 1)
• Range: 0 (perfect) to 1 (worst)
• Decomposition: BS = Reliability - Resolution + Uncertainty

A lower Brier score indicates better overall probabilistic predictions. Unlike ECE, it penalizes both overconfidence and poor discrimination, making it a comprehensive metric for probabilistic forecast quality.

A set of statistical techniques that enable a model to estimate the confidence of its own predictions. Unlike simple softmax probabilities, proper uncertainty quantification decomposes predictive uncertainty into aleatoric uncertainty (inherent data noise) and epistemic uncertainty (model ignorance). In legal AI, this allows a system to flag high-risk outputs for human review or trigger an abstention mechanism rather than generating a hallucinated response with false confidence.

• Bayesian Neural Networks learn a distribution over weights
• Monte Carlo Dropout approximates Bayesian inference at test time
• Deep Ensembles measure disagreement across independently trained models

A model-agnostic framework that generates prediction sets with a formal, finite-sample guarantee of coverage. Unlike calibration error metrics that measure average accuracy, conformal prediction provides a statistically rigorous method for controlling the error rate of a legal classifier on every individual prediction.

• Produces a set of plausible labels instead of a single point prediction
• Guarantees that the true label is included with a user-specified probability (e.g., 95%)
• Works with any pre-trained model without retraining
• Critical for high-stakes legal decisions where risk control is mandatory

A quantitative evaluation framework that measures the factual consistency of a generated summary or answer relative to the source material. While calibration error measures probabilistic alignment, faithfulness metrics directly detect contradictions and unsupported fabrications in the generated text itself.

• Entailment-based metrics use NLI models to check if each claim is supported
• Question-answering based metrics decompose text into atomic facts for verification
• Essential for detecting extrinsic hallucinations where a model invents facts not present in the provided context

The primary empirical metric for measuring calibration error. ECE partitions predictions into M equally-spaced confidence bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin. A perfectly calibrated model has an ECE of zero.

• Formula: ECE = Σ (|B_m|/n) × |acc(B_m) − conf(B_m)|
• Reliability Diagrams visually plot confidence against accuracy
• Modern LLMs often exhibit overconfidence with ECE values significantly above zero
• Temperature scaling is the most common post-hoc fix to reduce ECE

A simple yet effective post-hoc calibration method that adjusts the sharpness of a model's predicted probability distribution without altering its accuracy or rank ordering. A single scalar parameter T is learned on a held-out validation set to minimize the negative log-likelihood.

• T > 1: Softens probabilities, reducing overconfidence
• T < 1: Sharpens probabilities, increasing confidence separation
• Does not affect the model's argmax predictions or accuracy
• The most common baseline for neural network calibration due to its simplicity and effectiveness

A standard evaluation dataset specifically designed to measure a model's propensity to reproduce common human falsehoods and misconceptions. Unlike accuracy benchmarks, TruthfulQA tests for truthfulness by targeting questions where humans frequently hold incorrect beliefs, making it a direct probe of a model's calibration between confidence and factual reality.

• Contains 817 questions across 38 categories including law, health, and finance
• Evaluates both truthfulness and informativeness
• Reveals that larger models often become more confidently wrong on misconception-prone questions