Federated Uncertainty Quantification

A Bayesian approximation technique that applies dropout at inference time across federated nodes to generate multiple stochastic forward passes. Each client runs T stochastic passes on local data, producing a distribution of predictions rather than a single point estimate. The variance across passes quantifies epistemic uncertainty—the model's uncertainty about what it doesn't know. This is critical for flagging out-of-distribution pathology scans where the model should defer to a clinician. The global model's dropout masks remain synchronized across clients to ensure consistent uncertainty calibration.

A technique where M independent models are trained in parallel across the federated network, each initialized with different random seeds and potentially trained on different client subsets. At inference, predictions from all ensemble members are aggregated to produce a predictive distribution. The disagreement among ensemble members captures model uncertainty, while the average confidence captures data uncertainty. Federated ensembles are particularly robust to non-IID data distributions because each member may specialize in different regions of the heterogeneous data landscape.

A metric that measures the miscalibration gap between a model's predicted confidence and its empirical accuracy, computed in a privacy-preserving federated manner. Predictions are binned by confidence level, and the weighted absolute difference between accuracy and confidence is aggregated across clients using secure aggregation. A well-calibrated model should have ECE approaching zero—when it says it's 90% confident, it should be correct 90% of the time. High ECE in clinical settings can lead to dangerous overconfidence in incorrect diagnoses.

A distribution-free framework that produces prediction sets with a mathematically guaranteed coverage probability—e.g., 95% of true labels fall within the predicted set. Unlike Bayesian methods, conformal prediction makes no assumptions about data distribution, making it ideal for heterogeneous clinical data. In the federated setting, each client computes nonconformity scores on a local calibration set, and these scores are securely aggregated to determine a global threshold. The output is a prediction set rather than a single class, explicitly communicating ambiguity.

The task of identifying inference-time inputs that differ fundamentally from the federated training distribution. Techniques include:

• Energy-based models: Assigning an energy score to each input; low-energy samples are in-distribution
• Mahalanobis distance: Computing distance from class-conditional Gaussian distributions in feature space
• Gradient-based methods: Analyzing gradient norms during a pseudo-backward pass In federated settings, OOD detectors must be calibrated across heterogeneous client distributions without centralizing validation data, ensuring a rare disease presentation at one hospital isn't incorrectly flagged as anomalous.

A principled approach that places prior distributions over model weights and computes posterior distributions given observed data, providing full uncertainty quantification. In federated settings, this is implemented through:

• Variational inference: Approximating the posterior with a simpler distribution, with local ELBO computations aggregated via FedAvg
• Stochastic gradient Langevin dynamics (SGLD): Injecting Gaussian noise into gradient updates to sample from the posterior
• Federated Laplace approximation: Computing a local quadratic approximation of the posterior around the MAP estimate These methods provide theoretically grounded uncertainty estimates but are computationally more expensive than MC Dropout or ensembles.

A metric that measures the miscalibration between a model's predicted confidence and its actual accuracy. In a federated setting, ECE is computed by binning predictions across all nodes and comparing the average confidence in each bin to the observed accuracy. A perfectly calibrated model will have an ECE of zero, meaning a prediction made with 90% confidence is correct exactly 90% of the time. This is critical for clinical decision support where overconfident misdiagnoses can lead to adverse patient outcomes.

A distribution-free framework that generates prediction intervals with a mathematically guaranteed coverage probability. Unlike Bayesian methods, conformal prediction makes no assumptions about the underlying data distribution. In a federated context, the nonconformity scores are computed locally and aggregated to produce a global threshold. For a user-specified significance level (e.g., α = 0.1), the resulting intervals are guaranteed to contain the true value at least 90% of the time, providing rigorous uncertainty quantification for high-stakes clinical decisions.

Two fundamental types of uncertainty that federated models must disentangle:

• Epistemic Uncertainty: The reducible uncertainty stemming from a lack of knowledge or data. This is high in regions of sparse training data and can be reduced by collecting more representative samples. Captured by techniques like Monte Carlo Dropout and Deep Ensembles.
• Aleatoric Uncertainty: The irreducible uncertainty inherent in the data itself, such as measurement noise or overlapping class boundaries. Captured by models that output a predictive variance alongside the mean. Disentangling these is essential for knowing when to trust a model versus when to request a second opinion.

A technique for estimating epistemic uncertainty by training multiple independent models with different random initializations and aggregating their predictions. In a federated setup, ensembles can be constructed by training distinct models on different subsets of clients or by using different random seeds at each node. The variance across ensemble members' predictions provides a robust measure of model uncertainty. Unlike single-model methods, ensembles capture multi-modal posterior distributions and are less prone to collapsing to overconfident predictions on out-of-distribution inputs.

The task of identifying inference-time inputs that differ fundamentally from the federated training distribution. A model should flag these inputs for clinical review rather than silently producing potentially dangerous predictions. Techniques include:

• Density estimation on learned feature representations
• Energy-based models that assign higher scores to in-distribution samples
• Mahalanobis distance in the model's embedding space Effective OOD detection is critical when deploying models across diverse hospital systems where patient demographics and equipment may vary significantly from the training population.

The continuous monitoring of a deployed federated model's performance to identify degradation caused by concept drift or data drift in distributed input streams. Key metrics include:

• Federated Population Stability Index (PSI): Quantifies shifts in input feature distributions across institutions
• Federated Characteristic Stability Index (CSI): Tracks individual feature drift
• Prediction distribution monitoring: Detects shifts in the model's output distribution Drift detection is essential for maintaining clinical safety, as patient populations, treatment protocols, and diagnostic equipment evolve over time.