Federated Ensemble Learning

Each client trains its own model autonomously on local data without sharing raw patient records. This preserves data sovereignty and complies with regulations like HIPAA and GDPR.

• Models can use heterogeneous architectures (e.g., CNNs, transformers) suited to local data characteristics
• No gradient or weight sharing occurs during training
• Eliminates single points of failure in the training pipeline

A central aggregation mechanism combines outputs from diverse local models to produce a final prediction. Common strategies include:

• Majority voting: Each model casts a vote; the class with the most votes wins
• Weighted averaging: Models with higher local validation accuracy receive greater influence
• Stacking: A meta-learner is trained on local model outputs to learn optimal combination weights
• Bayesian model averaging: Incorporates uncertainty estimates from each ensemble member

Ensemble performance depends on model diversity across clients. Heterogeneous local data distributions naturally create diverse decision boundaries, making the ensemble more robust than any single model.

• Guards against overfitting to any single institution's patient population
• Reduces variance from noisy labels or outlier samples in individual datasets
• Provides built-in redundancy — if one local model fails, others compensate

Unlike weight-sharing federated averaging, ensemble methods transmit only predictions or logits — not model parameters. This dramatically reduces bandwidth requirements.

• Prediction vectors are typically orders of magnitude smaller than model weights
• Enables participation from resource-constrained edge devices
• Supports asynchronous updates — clients can contribute predictions on their own schedule

Ensemble methods inherently resist adversarial clients or corrupted local models. Since predictions are aggregated rather than gradients, malicious actors cannot easily poison the global model.

• Outlier predictions can be detected and excluded via trimmed averaging
• No single client can dominate the ensemble output
• Provides verifiable audit trails — each prediction contribution is traceable to its source institution

Federated ensembles are well-suited for high-stakes medical diagnosis where multiple expert opinions improve confidence. Example: a network of radiology departments independently trains tumor classifiers, and the ensemble's consensus flags cases for review.

• Mirrors multidisciplinary team (MDT) workflows in clinical practice
• Each local model can specialize in subpopulation-specific patterns
• Ensemble disagreement signals can trigger human-in-the-loop review

The foundational aggregation algorithm that combines local model weights into a global model by computing a weighted average proportional to each client's dataset size. In ensemble contexts, FedAvg serves as the baseline against which more sophisticated fusion strategies are compared.

• Weights aggregated via stochastic gradient descent averaging
• Communication rounds synchronize local updates with the central server
• Assumes IID data distributions across clients
• Serves as the backbone for many ensemble initialization strategies

A distributed adaptation of Bootstrap Aggregating where each client trains a local model on its naturally occurring data partition, and predictions are combined via majority voting or averaging. The inherent data isolation across institutions creates natural bootstrap samples without resampling.

• Each hospital's dataset acts as a natural bootstrap replicate
• Reduces variance without increasing bias
• Particularly effective when local datasets are small but diverse
• Final ensemble prediction via weighted voting based on local validation performance

A two-level ensemble architecture where local models serve as base learners and a meta-learner is trained to optimally combine their predictions. In federated settings, the meta-learner can be trained on a public validation dataset without accessing private patient records.

• Level-0: Independent local models trained on site-specific clinical data
• Level-1: Meta-model learns to weight each local model's contribution
• Prevents overfitting to any single institution's data distribution
• Common meta-learners include logistic regression and gradient-boosted trees

A probabilistic fusion strategy that weights each local model's prediction by its posterior probability of being the correct model given the observed data. Unlike simple averaging, BMA explicitly accounts for model uncertainty in the aggregation process.

• Weights derived from marginal likelihood of each local model
• Naturally handles heterogeneous model architectures across clients
• Provides well-calibrated uncertainty estimates for clinical decision support
• Computationally intensive but offers superior robustness to distribution shift

Techniques that explicitly encourage local models to learn complementary features rather than converging to similar solutions. Diversity is the critical ingredient that makes ensembles outperform individual models.

• Negative correlation learning penalizes models that make similar errors
• Dropout-based diversity randomly masks different neurons per client
• Adversarial training encourages models to specialize in different data regions
• Measured via Q-statistic and kappa-error diagrams to quantify ensemble complementarity

A fusion approach where the collective knowledge of multiple local teacher models is distilled into a single student model using soft probability outputs rather than hard labels. The student learns the dark knowledge of the ensemble without inheriting its computational cost.

• Teachers share logits on a public distillation dataset, not model weights
• Student trained via temperature-scaled softmax matching
• Compresses ensemble into a single deployable model
• Particularly valuable for edge deployment where ensemble inference is impractical