Confidence Calibration

A visual tool for diagnosing calibration errors. It plots predicted probability against observed accuracy.

• Perfect Calibration: Points fall on the diagonal y=x line.
• Underconfidence: Curve sits above the diagonal (model is more accurate than it thinks).
• Overconfidence: Curve sits below the diagonal (model is less accurate than its high probabilities suggest).

This is the primary diagnostic for identifying systematic bias in a relation extraction model's probability outputs.

The primary scalar metric for measuring miscalibration. ECE partitions predictions into M bins and computes a weighted average of the absolute difference between accuracy and confidence in each bin.

• Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|
• Lower is better: An ECE of 0 indicates perfect calibration.
• Limitation: ECE is sensitive to binning strategy and can be gamed by predicting the base rate for all instances.

A post-hoc calibration method that fits a logistic regression model on top of a classifier's raw output scores (logits).

• Learns two scalar parameters (A, B) to transform scores: q_i = σ(A * z_i + B)
• Best for: Binary relation classification and small calibration datasets.
• Assumption: Calibration error is a smooth, sigmoidal function of the raw score.

Originally developed for Support Vector Machines, it remains a robust baseline for calibrating neural relation extractors.

A non-parametric calibration method that learns a piecewise constant, monotonically increasing mapping from raw scores to calibrated probabilities.

• Advantage: Makes no assumptions about the functional form of the miscalibration.
• Disadvantage: Prone to overfitting on small datasets; requires a dedicated hold-out calibration set.
• Use case: Ideal when the reliability diagram shows a complex, non-sigmoidal distortion in predicted probabilities for multi-class relation extraction.

A simple yet highly effective calibration method for neural networks. It uses a single scalar parameter T (temperature) to soften or sharpen the output softmax distribution.

• Mechanism: q_i = softmax(z_i / T)
• T > 1: Softens probabilities, reducing overconfidence.
• T < 1: Sharpens probabilities, reducing underconfidence.
• Key benefit: T is optimized on a validation set using negative log-likelihood, but does not alter the model's accuracy or rank-ordering of predictions.

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and actual binary outcomes.

• Formula: BS = (1/N) * Σ (p_i - y_i)²
• Decomposition: Can be broken down into refinement loss and calibration loss.
• Comparison to ECE: Unlike ECE, the Brier Score is a proper scoring rule, meaning its optimal value is achieved only by the true conditional probability distribution. It penalizes both poor discrimination and poor calibration simultaneously.

The primary empirical metric for measuring miscalibration. ECE discretizes predicted probabilities into bins and computes a weighted average of the absolute difference between accuracy and confidence within each bin.

• A perfectly calibrated model has an ECE of 0.
• Commonly visualized using reliability diagrams.
• A low ECE indicates that when a model says it's 80% confident, it is correct roughly 80% of the time.

A parametric method for calibrating model outputs by fitting a logistic regression model on top of the original classifier's scores. It learns scalar parameters to map raw logits to well-calibrated probabilities.

• Works best when miscalibration is sigmoidal.
• Requires a held-out validation set to prevent overfitting.
• More effective than Isotonic Regression for small calibration datasets.

A non-parametric calibration method that fits a piecewise constant, monotonically increasing function to map model scores to calibrated probabilities. It makes no assumptions about the functional form of the miscalibration.

• More flexible than Platt Scaling but prone to overfitting on small datasets.
• The resulting mapping is a step function.
• Ideal when the model's raw confidence scores are not monotonically related to accuracy.

A simple and highly effective calibration technique for neural networks, particularly distilled models. It divides all logits by a single scalar parameter, the temperature (T), before applying the softmax function.

• A T > 1 softens the probability distribution, reducing overconfidence.
• Does not affect the model's accuracy or rank ordering of predictions.
• The optimal T is learned by minimizing NLL on a validation set.

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and the actual binary outcomes. It jointly evaluates both discrimination and calibration.

• Decomposable into a refinement term and a calibration term.
• A lower Brier score indicates better overall probabilistic predictions.
• Unlike ECE, it is a continuous metric sensitive to the magnitude of probability errors.

The downstream task that directly benefits from calibrated confidence scores. RE identifies semantic relationships like 'founded_by' or 'acquired' between entities in text. Calibrated probabilities allow a system to set reliable extraction thresholds.

• Prevents populating a knowledge graph with low-confidence, noisy triples.
• Essential for distant supervision, where noisy labels require robust confidence estimates.
• Enables risk-aware decision making in automated pipelines.