Model Calibration

The Expected Calibration Error (ECE) is the most common scalar summary of miscalibration. It approximates the average difference between a model's confidence and its accuracy.

• Calculation: Predictions are binned by confidence score (e.g., 0.0-0.1, 0.1-0.2). ECE is a weighted average of the absolute difference between the average confidence and average accuracy within each bin.
• Federated Context: Computing ECE requires aggregating confidence-accuracy pairs from all clients, which must be done without centralizing raw data, often via secure aggregation of histogram statistics.
• Limitation: The binning process can obscure miscalibration within bins. The number of bins is a hyperparameter.

The Maximum Calibration Error (MCE) measures the worst-case miscalibration observed across all confidence bins. It is defined as the maximum absolute difference between bin confidence and bin accuracy.

• Use Case: Crucial for safety-critical applications (e.g., medical diagnosis, autonomous systems) where even a single severely miscalibrated prediction could have catastrophic consequences.
• Interpretation: A low MCE indicates no bin has dangerously high miscalibration, even if the average ECE is acceptable.
• Federated Challenge: Identifying the maximum error requires a global view of all bins, again necessitating privacy-preserving aggregation of client histograms.

The Adaptive Calibration Error (ACE) is a variant of ECE that uses bins containing an equal number of samples, rather than bins of equal confidence width.

• Advantage: Mitigates a key flaw in ECE where bins may have very few samples, making the accuracy estimate within that bin unreliable. ACE ensures each bin's accuracy is estimated from a sufficient number of data points.
• Federated Benefit: The equal-sample approach can be more stable when data is non-uniformly distributed across clients, a common scenario in federated learning with non-IID data.

The Brier Score is a proper scoring rule that measures the mean squared error between the predicted probability and the actual outcome (0 or 1). It decomposes into two components: calibration loss and refinement loss.

• Decomposition: Brier Score = Calibration Loss + Refinement Loss. Calibration loss directly measures miscalibration, while refinement loss captures the model's ability to separate classes (akin to sharpness).
• Holistic View: A single metric that jointly evaluates calibration and discrimination. A lower Brier Score is better.
• Federated Computation: Can be computed by securely aggregating the sum of squared errors from each client, as it relies on a simple average.

The Negative Log-Likelihood (NLL), or cross-entropy loss, is the primary training objective for probabilistic classification models. It also serves as a calibration metric.

• Mechanism: Heavily penalizes confident but incorrect predictions. A well-calibrated model minimizing NLL will naturally assign high probability to correct labels and low probability to incorrect ones.
• Relation to ECE: While NLL is sensitive to miscalibration, a model can have good NLL but poor ECE if errors are systematic across confidence levels. They are complementary measures.
• Federated Baseline: The global model's NLL on a held-out test set is a standard federated evaluation metric, computed via loss aggregation from clients.

A Reliability Diagram is a visual diagnostic tool, not a single metric. It plots the observed accuracy (y-axis) against the predicted confidence (x-axis) for a set of bins.

• Interpretation: A perfectly calibrated model yields a diagonal line (accuracy = confidence). Deviations below the diagonal indicate overconfidence; deviations above indicate underconfidence.
• Federated Application: Essential for understanding how a model is miscalibrated across different sub-populations (clients). Generating a global diagram requires aggregating bin statistics from all clients in a privacy-preserving manner.
• Actionable Insight: Guides interventions like temperature scaling or label smoothing, which can be applied during the federated aggregation phase.

Expected Calibration Error (ECE) is the primary quantitative metric for measuring model calibration. It computes the average absolute difference between a model's predicted confidence and its empirical accuracy, typically by grouping predictions into bins based on confidence scores.

• Calculation: ECE = Σ (|acc(b) - conf(b)| * n_b / N) across bins.
• Interpretation: A lower ECE indicates better calibration. A perfectly calibrated model has an ECE of 0.
• Federated Context: Computing ECE requires aggregating predictions and labels across clients, which must be done via secure aggregation or federated evaluation protocols to preserve privacy.

A Reliability Diagram is a visual diagnostic tool for assessing model calibration. It plots a model's predicted confidence (on the x-axis) against its observed empirical accuracy (on the y-axis).

• Perfect Calibration: Points fall on the diagonal y=x line.
• Diagnosis: Points below the diagonal indicate overconfidence; points above indicate underconfidence.
• Use Case: In federated learning, a global reliability diagram can be constructed from aggregated client statistics, while local diagrams can reveal client drift and data heterogeneity issues.

Platt Scaling is a post-processing calibration technique that fits a logistic regression model to a model's raw output scores (logits) to produce better-calibrated probabilities.

• Process: Uses a held-out validation set to learn two parameters that scale and shift the logits.
• Application: Commonly used for Support Vector Machines and neural networks. In federated settings, it can be applied locally on client devices or learned globally via federated optimization on aggregated validation statistics.
• Limitation: Assumes the miscalibration is sigmoid-shaped, which may not hold for all models.

Temperature Scaling is a lightweight, single-parameter variant of Platt Scaling used specifically for neural networks. It softens or sharpens the output probability distribution by dividing the logits by a learned temperature parameter T.

• Effect: T > 1 softens probabilities (reduces confidence), T < 1 sharpens them.
• Advantage: Preserves the predicted class ranking while improving calibration. It is computationally efficient and well-suited for on-device evaluation and deployment.
• Federated Learning: The temperature parameter T can be optimized centrally or via federated hyperparameter tuning.

The Brier Score is a proper scoring rule that measures the mean squared error between predicted probabilities and the actual outcomes (0 or 1). It decomposes into calibration loss and refinement loss.

• Formula: BS = (1/N) Σ (p_i - o_i)², where p_i is the predicted probability and o_i is the actual binary outcome.
• Interpretation: A lower Brier Score is better. It simultaneously penalizes both miscalibration and inaccuracy.
• Federated Relevance: Like ECE, it requires label aggregation for calculation, making it a candidate for federated evaluation frameworks that ensure privacy.

Maximum Calibration Error (MCE) is a calibration metric that focuses on the worst-case miscalibration across all confidence bins. It reports the maximum absolute difference between accuracy and confidence.

• Calculation: MCE = max_b |acc(b) - conf(b)| across all bins.
• Use Case: Critical in high-stakes applications (e.g., healthcare, finance) where even a single region of severe overconfidence is unacceptable. It provides a safety-focused, conservative measure.
• Relation to ECE: While ECE gives an average, MCE highlights the peak error, complementing the overall assessment of model reliability.