Confidence Calibration

The primary metric for measuring miscalibration. ECE partitions predictions into bins by confidence score and computes the weighted absolute difference between average confidence and accuracy within each bin.

• Perfect Calibration: ECE = 0
• Calculation: Weighted average of |accuracy(B) - confidence(B)|
• Visual Tool: Reliability diagrams plot this relationship directly

A model with 95% confidence but only 80% accuracy in that bin has a 15% calibration gap.

A post-hoc calibration method that learns a single scalar parameter T (temperature) to soften or sharpen output probabilities without altering the model's ranking of predictions.

• T > 1: Softens probabilities, reducing overconfidence
• T < 1: Sharpens probabilities, reducing underconfidence
• Optimization: T is learned on a held-out validation set using negative log-likelihood

This is the simplest and most widely used extension of Platt scaling for modern neural networks.

Modern deep neural networks, especially large language models, are systematically overconfident—their predicted probabilities are higher than their empirical accuracy.

• Root Cause: Training with cross-entropy loss on one-hot labels pushes probabilities toward extremes
• Consequence: A model may assign 99% confidence to incorrect predictions, undermining trust in high-stakes applications
• Mitigation: Label smoothing, temperature scaling, and Bayesian approximations can reduce this gap

A visual diagnostic tool that plots expected accuracy against predicted confidence. In a perfectly calibrated model, the plot forms the identity line y = x.

• Above the diagonal: Model is underconfident (accuracy exceeds confidence)
• Below the diagonal: Model is overconfident (confidence exceeds accuracy)
• Gap size: The area between the curve and the diagonal represents total miscalibration

These diagrams provide an immediate visual assessment of calibration quality across all confidence levels.

A proper scoring rule that measures the mean squared difference between predicted probability and the actual binary outcome. It jointly evaluates discrimination and calibration.

• Formula: (1/N) * Σ(prediction - outcome)²
• Range: 0 (perfect) to 1 (worst)
• Decomposition: Can be broken into refinement loss and calibration loss

Unlike ECE, the Brier score penalizes both overconfidence and poor separation between classes.

A non-parametric calibration method that learns a monotonically increasing piecewise constant function to map uncalibrated scores to calibrated probabilities.

• Advantage: Makes no assumptions about the shape of the miscalibration curve
• Trade-off: Requires more calibration data than parametric methods like temperature scaling
• Application: Effective when the model's confidence distortion is complex and non-linear

This method is particularly useful when reliability diagrams show irregular, non-smooth miscalibration patterns.

A quantitative evaluation score measuring the degree to which a generated statement is logically entailed by the provided source context. Unlike calibration, which measures a model's self-assessment, faithfulness measures objective alignment with evidence.

• Uses Natural Language Inference (NLI) to detect contradictions
• Independent of the model's own confidence score
• Critical for catching high-confidence hallucinations

A technique where a language model generates an initial response, then systematically drafts and answers independent verification questions to self-correct factual errors. This process directly improves a model's ability to recognize when it is likely wrong.

• Reduces the miscalibration gap between confidence and accuracy
• Each verification step targets a specific atomic claim
• Enables self-supervised improvement without external tools

A binary or graded evaluation step that verifies whether every atomic claim in a generated response can be traced back to the specific context provided. A perfectly calibrated model that is poorly grounded will confidently assert unsupported information.

• Operates at the claim level, not the response level
• Essential for detecting intrinsic hallucinations
• Often implemented via NLI models or LLM-as-a-judge

A dynamic or static metric assigned to a data source based on factors like historical accuracy, domain authority, and content freshness. Calibration must account for source quality—a 90% confidence score on a low-reliability source should be interpreted differently than the same score on a verified authority.

• Used to weight evidence during retrieval
• Enables confidence discounting for untrusted sources
• Combines with temporal grounding for freshness checks

A classification system categorizing factual errors into distinct types: intrinsic hallucinations (contradicting provided context) vs. extrinsic hallucinations (unsupported by any source). Calibration systems must be trained to recognize and flag both categories.

• Source-conflict errors: when multiple sources disagree
• Temporal misalignment: outdated facts presented as current
• Enables targeted mitigation strategies per error type

A grounding strategy requiring multiple independent retrieved documents to corroborate a fact before it is presented as true. This directly impacts calibration by providing a stronger signal for confidence estimation.

• Reduces reliance on any single potentially erroneous source
• Confidence scores can be boosted when corroboration threshold is met
• Mitigates source-conflict hallucinations through consensus