Calibration Layer

The primary diagnostic tool for assessing calibration quality. A reliability diagram plots predicted probability against observed frequency by binning predictions into intervals (e.g., 0-10%, 10-20%) and computing the true fraction of positives in each bin. A perfectly calibrated model follows the identity diagonal line. Deviations reveal systematic miscalibration:

• Above diagonal: Model is underconfident (predicts lower probability than reality)
• Below diagonal: Model is overconfident (predicts higher probability than reality)
• S-shaped curve: Indicates the model's scores are too concentrated at extremes

A scalar summary metric that quantifies the weighted average gap between confidence and accuracy across all prediction bins. ECE is calculated by partitioning predictions into M bins, computing the absolute difference between average confidence and accuracy per bin, and weighting each bin by its sample proportion. Lower ECE indicates better calibration. A typical fraud detection system targets ECE below 0.05. ECE complements reliability diagrams by providing a single number for automated monitoring, enabling drift detection when calibration degrades in production due to shifting transaction patterns.

A single-parameter variant of Platt Scaling widely used for calibrating neural network outputs. It divides the logits (pre-softmax values) by a learned temperature parameter T before applying softmax. When T > 1, the output distribution softens, reducing overconfidence. When T < 1, it sharpens. The key advantage is that temperature scaling preserves the rank order of predictions and does not alter accuracy—only confidence estimates. It is optimized by minimizing negative log-likelihood on a held-out validation set, making it computationally lightweight and ideal for deep learning fraud classifiers.

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and actual binary outcomes. Calculated as BS = (1/N) * Σ(p_i - y_i)², where p_i is the predicted probability and y_i is the ground truth (0 or 1). The Brier Score decomposes into three components:

• Refinement: How sharp the predictions are
• Calibration: How well probabilities match outcomes
• Uncertainty: Inherent difficulty of the prediction task A lower Brier Score indicates both better calibration and discrimination, making it a holistic metric for evaluating fraud probability estimates.

A parametric calibration method that fits a logistic regression model to the raw scores of a classifier. It works by finding the optimal parameters A and B to transform scores into probabilities using the sigmoid function: P(y=1|score) = 1 / (1 + exp(A*score + B)). Platt Scaling is most effective when the uncalibrated score distribution is sigmoid-shaped and assumes a specific functional form, making it computationally efficient but less flexible than non-parametric alternatives.

A non-parametric calibration technique that learns a monotonically increasing step function to map raw scores to calibrated probabilities. Unlike Platt Scaling, it makes no assumptions about the shape of the mapping function, allowing it to correct any monotonic distortion. The trade-off is a higher risk of overfitting, especially with small datasets, and it requires careful regularization to prevent probability estimates from becoming unstable in sparse regions of the score space.

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and actual binary outcomes. Calculated as (1/N) * Σ(predicted - actual)², a lower Brier Score indicates better calibration. It decomposes into refinement loss and calibration loss, allowing practitioners to distinguish whether poor performance stems from a lack of discriminative power or miscalibrated probability estimates. Essential for comparing calibration methods objectively.

A visual diagnostic tool that plots observed frequency against predicted probability across discrete bins. A perfectly calibrated model follows the diagonal identity line. Systematic deviations reveal specific miscalibration patterns:

• Above the diagonal: Underconfidence (predicted probabilities too low)
• Below the diagonal: Overconfidence (predicted probabilities too high)
• S-shaped curve: Typical of uncalibrated maximum margin classifiers like SVMs

A single-parameter variant of Platt Scaling widely used in neural network calibration. It divides all logits by a learned temperature parameter T before applying softmax: softmax(logits / T). When T > 1, the output distribution softens and entropy increases, correcting overconfidence. Unlike full Platt Scaling, it preserves the relative ranking of predictions and does not alter accuracy, making it ideal for calibrating deep ensembles and large language model confidence scores.

A summary statistic that quantifies miscalibration by computing the weighted average of the absolute difference between accuracy and confidence across M bins: ECE = Σ(|B_m|/n) * |acc(B_m) - conf(B_m)|. While widely used, ECE has known limitations:

• Sensitive to binning strategy (equal-width vs. equal-mass)
• Can be misleading with imbalanced datasets
• Does not distinguish underconfidence from overconfidence Use alongside reliability diagrams for complete diagnosis.