Calibration Score

The primary metric for measuring miscalibration. ECE partitions predictions into M equally-spaced confidence bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin.

• Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|
• Perfect calibration: ECE = 0
• Lower ECE indicates better alignment between predicted probabilities and observed frequencies
• Commonly visualized using reliability diagrams that plot accuracy against confidence

A graphical diagnostic tool that visually exposes calibration errors. The diagram plots observed accuracy on the y-axis against predicted confidence on the x-axis.

• A perfectly calibrated model follows the diagonal identity line (y = x)
• Underconfidence: points lie above the diagonal — the model is more accurate than it claims
• Overconfidence: points lie below the diagonal — the model is less accurate than its predictions suggest
• Gap analysis between the curve and diagonal reveals systematic bias in probability estimates

A post-hoc calibration technique that applies a single scalar parameter T (temperature) to the logits before the softmax function, without altering the model's accuracy or ranking.

• T > 1: Softens probabilities, reducing overconfidence
• T < 1: Sharpens probabilities, reducing underconfidence
• T = 1: Original uncalibrated probabilities
• Optimized on a held-out validation set using negative log-likelihood
• Preserves AUC and accuracy while dramatically improving ECE

A parametric calibration method that fits a logistic regression model on top of a classifier's raw scores, transforming them into well-calibrated posterior probabilities.

• Learns two scalar parameters A and B such that P(y=1|x) = 1 / (1 + exp(A * f(x) + B))
• Originally developed for Support Vector Machines but widely applicable
• Works best when miscalibration is sigmoid-shaped in reliability diagrams
• Requires a separate calibration dataset distinct from training data to avoid overfitting

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and actual binary outcomes. Unlike ECE, it decomposes into refinement and calibration components.

• Formula: BS = (1/N) Σ (p_i - o_i)² where p_i is predicted probability and o_i is the actual outcome
• Range: 0 (perfect) to 1 (worst)
• Penalizes both discrimination (separation of classes) and calibration
• A lower Brier score indicates both accurate and well-calibrated predictions

A non-parametric calibration method that learns a monotonically increasing piecewise constant function mapping uncalibrated scores to calibrated probabilities, making no assumptions about the shape of miscalibration.

• More flexible than Platt scaling — can correct any monotonic distortion
• Prone to overfitting on small datasets due to high capacity
• Best suited for large validation sets where parametric assumptions fail
• Produces step-function transformations visible in calibrated reliability curves

A metric quantifying how well a generated statement is supported by a specific source document or verified knowledge base. Unlike calibration score—which measures a model's self-assessed confidence—grounding score measures external factual anchoring. A high grounding score indicates that claims can be directly attributed to provided context, making it essential for Retrieval-Augmented Generation (RAG) pipelines where hallucination prevention depends on source fidelity.

A quantitative score measuring the degree to which a generated summary or answer contains only claims directly inferable from the source document. This metric detects extrinsic hallucinations—statements that sound plausible but have no evidentiary basis in the provided context. Faithfulness is often computed using Natural Language Inference (NLI) models that classify each generated claim as entailed, contradicted, or neutral relative to the source.

A Natural Language Inference task that determines whether a generated hypothesis statement logically follows from a given premise text. Used as a binary gate in content pipelines:

• Entailment: The claim is supported; output passes.
• Contradiction: The claim conflicts with the source; output is blocked.
• Neutral: The claim is unrelated; may trigger human review. This check is a core building block for automated fact-verification systems.

A threshold-based filter that compares vector embeddings of generated text against a reference source. If the cosine similarity falls below a predefined minimum, the output is blocked or flagged. This guard operates at the semantic level rather than exact string matching, allowing it to detect subtle meaning drift. Commonly deployed in production RAG systems to catch responses that stray from retrieved context even when individual words appear relevant.

The frequency at which a language model generates factually incorrect, nonsensical, or unfaithful output not grounded in its training data or provided context. Measured as a percentage of total outputs in evaluation runs. While calibration score tells you whether the model's confidence aligns with its accuracy, hallucination rate tells you how often it fabricates entirely. Both metrics together provide a complete picture of output trustworthiness.

A system that tracks the gradual shift in meaning or contextual relevance of generated content over time. In programmatic content pipelines, models may slowly diverge from their intended topic distribution due to prompt decay or data distribution shifts. The monitor uses embedding-space trajectory analysis to detect when output clusters migrate away from reference centroids, alerting operators before content quality degrades measurably.