Expected Calibration Error

ECE quantifies the mismatch between confidence and accuracy by partitioning predictions into M equally-spaced bins and computing a weighted average of the absolute difference between accuracy and confidence within each bin.

• Formula: ECE = Σ (|B_m| / n) · |acc(B_m) - conf(B_m)|
• Range: [0, 1], where 0 indicates perfect calibration
• Bin count M: Typically 10 or 15 bins for stable estimation
• A perfectly calibrated model predicting 80% confidence should be correct exactly 80% of the time

ECE is most commonly interpreted through a reliability diagram, which plots expected sample accuracy against confidence.

• Perfect calibration: All points lie on the diagonal identity line
• Overconfidence: Points fall below the diagonal — model is too sure of wrong answers
• Underconfidence: Points fall above the diagonal — model is more accurate than it believes
• The gap area between the curve and the diagonal visually represents the ECE value

ECE can be decomposed into two orthogonal components that separate calibration error from inherent predictive power:

• Calibration Error: The actual miscalibration — can be reduced by post-hoc scaling without changing model accuracy
• Refinement Error: The irreducible error determined by how well the model separates classes; related to the Brier score decomposition
• This decomposition ensures that a model with poor accuracy but perfect calibration can still achieve ECE = 0

Despite its widespread use, ECE has known failure modes that practitioners must account for:

• Bin sensitivity: Results vary with the number and spacing of bins; fixed-width binning can produce empty bins in low-data regimes
• Class imbalance blindness: ECE can be low for a model that predicts the majority class with high confidence while ignoring minority classes entirely
• Adaptive binning: Equal-mass binning schemes mitigate empty-bin issues but can obscure overconfidence in sparse regions
• Static metric: ECE does not capture distributional shift over time — a model calibrated on training data may not remain calibrated in deployment

ECE is the target metric optimized by post-hoc calibration techniques that adjust model outputs without retraining:

• Temperature Scaling: A single scalar parameter T > 0 divides all logits before softmax; T > 1 flattens distributions, reducing overconfidence
• Platt Scaling: Fits a logistic regression model on top of raw classifier scores
• Isotonic Regression: A non-parametric method that learns a monotonic mapping from scores to calibrated probabilities
• Beta Calibration: A parametric family specifically designed for binary classifiers that generalizes Platt scaling

In mission-critical RF domains, ECE directly impacts operational trust and decision thresholds:

• Signal classification: An overconfident automatic modulation classifier may misclassify a hostile radar as benign with 99% reported confidence
• Spectrum sensing: Calibrated uncertainty enables optimal Neyman-Pearson detection thresholds for cognitive radios
• Adversarial robustness: ECE often degrades sharply under adversarial perturbation, serving as a detection signal for attacks
• Out-of-distribution detection: Well-calibrated models naturally assign lower confidence to unknown signal types, improving OOD rejection

A visual diagnostic tool that plots observed accuracy against predicted confidence by binning predictions into equal-width intervals. A perfectly calibrated model follows the identity line (y=x). Deviations above the line indicate underconfidence, while deviations below reveal overconfidence. In RFML, this is essential for visualizing whether a signal classifier's reported 90% confidence in identifying a specific modulation scheme actually corresponds to 90% empirical accuracy in that bin.

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and actual binary outcomes. Unlike ECE, which bins predictions, the Brier Score provides a continuous, differentiable measure of both discrimination and calibration quality. For multi-class RF classification problems, the multi-class Brier Score decomposes into uncertainty, resolution, and reliability components, offering a more granular view of model performance than accuracy alone.

A post-hoc calibration method that applies a single scalar parameter T to soften or sharpen the output logits of a neural network before the softmax function. When T > 1, the probability distribution is softened, reducing overconfidence. When T < 1, it is sharpened. The optimal temperature is learned by minimizing the negative log-likelihood on a held-out validation set. This technique is widely adopted in RFML because it does not alter classification accuracy while significantly reducing ECE.

A parametric calibration method that fits a logistic regression model to the raw classifier scores. Originally developed for Support Vector Machines, it maps uncalibrated outputs to well-calibrated probabilities using a learned sigmoid function. In RFML applications, Platt Scaling is effective when the calibration error is primarily due to systematic bias rather than complex distributional mismatches, making it suitable for legacy signal processing pipelines being retrofitted with confidence estimation.

A non-parametric calibration method that learns a monotonically increasing piecewise constant function mapping raw scores to calibrated probabilities. Unlike Platt Scaling or Temperature Scaling, it makes no assumptions about the functional form of the miscalibration. This flexibility is valuable in RFML when dealing with non-linear confidence distortions caused by complex channel impairments. However, it requires more calibration data and is prone to overfitting on small datasets.

A metric that weights each correct prediction by its associated confidence score, penalizing models that are confidently wrong more heavily than those that are hesitantly correct. In mission-critical RFML applications like emitter identification, a high-confidence misclassification of an adversarial radar can trigger catastrophic downstream decisions. This metric directly quantifies the operational risk of miscalibration beyond what ECE captures through simple binning.