Expected Calibration Error (ECE)

ECE measures the weighted average difference between a model's predicted confidence and its actual accuracy within discrete probability bins. The process involves:

• Partitioning all predictions into M equally-spaced confidence bins (e.g., 0-0.1, 0.1-0.2)
• Computing the accuracy gap for each bin: |accuracy(B_m) - confidence(B_m)|
• Calculating the weighted sum of these gaps, where each bin's weight is the proportion of samples falling into it A perfectly calibrated model has an ECE of 0, meaning a prediction with 90% confidence is correct exactly 90% of the time.

In a federated setting, ECE is computed without centralizing raw predictions or labels. The process uses secure aggregation of bin statistics:

• Each client independently bins its local predictions and computes bin-specific counts and accuracy sums
• Only these aggregate statistics—not individual predictions—are transmitted to the central server
• The server sums the counts and accuracies across all clients to reconstruct the global calibration curve This preserves patient privacy while providing a comprehensive view of model reliability across the entire distributed network.

ECE is typically visualized through a reliability diagram (calibration plot), which plots:

• X-axis: Mean predicted confidence within each bin
• Y-axis: Observed empirical accuracy within each bin
• Diagonal reference line: Represents perfect calibration Deviations below the diagonal indicate overconfidence (the model is less accurate than it claims), while deviations above indicate underconfidence. In federated evaluation, this diagram can be reconstructed from the aggregated bin statistics without exposing any single institution's patient outcomes.

For high-stakes medical applications, calibration is often more critical than raw accuracy. A miscalibrated diagnostic model can lead to:

• Overconfident misdiagnoses: A model assigning 99% probability to an incorrect cancer diagnosis erodes clinician trust and may lead to unnecessary interventions
• Underconfident correct predictions: A model hedging on a correct diagnosis with only 60% confidence may cause clinicians to disregard valuable AI assistance ECE provides a single interpretable scalar that regulatory bodies and clinical governance boards can use to audit model safety before deployment.

Standard ECE has known weaknesses that are particularly relevant in federated healthcare settings:

• Bin sensitivity: The choice of bin count M affects the metric; too few bins mask miscalibration, too many produce noisy estimates
• Class imbalance blindness: ECE can appear low for a model that simply predicts the majority class with high confidence
• Adaptive binning schemes like Equal-Mass ECE or Maximum Calibration Error (MCE) address these issues by dynamically sizing bins or focusing on the worst-case deviation
• Federated ECE must also account for label distribution skew across institutions, which can distort global calibration estimates if not properly weighted.

ECE is closely related to group fairness and uncertainty quantification metrics evaluated in federated settings:

• Federated Demographic Parity and Equalized Odds can be decomposed into calibration and accuracy components, revealing whether unfairness stems from miscalibration or differing base rates
• Federated Conformal Prediction builds on calibration principles to produce prediction sets with guaranteed coverage, offering a practical alternative to raw probability estimates
• Federated Uncertainty Quantification methods like Deep Ensembles directly aim to improve ECE by better modeling epistemic uncertainty across heterogeneous data sources.