Federated Ensemble

Unlike Federated Averaging (FedAvg) which merges parameters, federated ensembles train each client model to completion locally. Each hospital retains a model optimized for its unique patient demographics, imaging protocols, and label distributions. The global model is a meta-learner or aggregation function that combines predictions, not weights.

• Preserves site-specific clinical expertise
• No requirement for identical model architectures across clients
• Naturally handles feature distribution skew and label distribution skew

The ensemble combines outputs rather than parameters. Common strategies include majority voting for classification, weighted averaging of predicted probabilities, or training a stacking meta-model on a validation set. This avoids the weight divergence problem that plagues parameter-averaging methods under non-IID conditions.

• Majority voting: Each local model casts a vote; the class with the most votes wins
• Weighted averaging: Client predictions are weighted by local validation performance
• Stacking: A trainable combiner learns which clients to trust for which inputs

Because only predictions are shared, each institution can use a model architecture suited to its computational resources and data modalities. A rural clinic might deploy a lightweight MobileNet while an academic medical center uses a Vision Transformer. This heterogeneous model zoo is impossible under parameter-averaging federated learning.

• Enables cross-institutional collaboration without infrastructure mandates
• Supports multi-modal fusion where different sites contribute different data types
• Reduces barriers to entry for resource-constrained participants

Federated ensembles exhibit natural robustness to Byzantine failures and adversarial clients. A malicious participant submitting corrupted model updates can poison a FedAvg global model. In an ensemble, a compromised client produces only corrupted predictions, which can be identified as outliers and excluded by the aggregation mechanism.

• Outlier detection on prediction distributions flags anomalous clients
• No single point of failure in the aggregation process
• Compatible with differential privacy guarantees at the prediction level

Federated knowledge distillation is a closely related paradigm where clients share soft labels (predicted class probabilities) on a public, unlabeled dataset rather than model parameters. A central student model is trained to mimic the ensemble's collective predictions. This achieves the benefits of ensembling while producing a single deployable model.

• Public dataset can be unlabeled, open-source medical text or imagery
• Soft labels convey richer information than hard classifications
• Student model distills the collective intelligence of the federated network

In Federated Prototype Learning, clients share compact class-representative vectors (prototypes) instead of full models or predictions. The global ensemble computes distances between a query sample and all client prototypes. This is highly communication-efficient and naturally robust to label distribution skew, as prototypes capture local class geometry.

• Prototypes are privacy-preserving by design—they are aggregated class summaries
• Communication cost is orders of magnitude lower than sharing model weights
• Effective for few-shot learning scenarios in rare disease diagnosis

A privacy-preserving alternative to parameter averaging where clients share soft label predictions on a public dataset instead of model weights. This enables heterogeneous model architectures across institutions—each hospital can use its preferred neural network design while still contributing to collective intelligence. The ensemble of teacher models generates consensus predictions that train a compact student model, naturally handling label distribution skew without requiring identical model topologies.

A communication-efficient method where clients share compact class-representative vectors (prototypes) instead of full model updates. Each client computes the mean embedding for each class locally and transmits only these statistical summaries. The global server aggregates prototypes to form decision boundaries, naturally handling label distribution skew because prototypes are class-conditional and invariant to sample count imbalances. This approach provides inherent privacy guarantees since raw data never leaves local sites.

A strategy that partitions clients into groups with similar data distributions and trains a separate model for each cluster. This directly addresses extreme statistical heterogeneity where a single global model would fail. Clustering can be based on:

• Model weight similarity after local training
• Performance cross-validation between clients
• Explicit metadata about patient demographics Each cluster forms its own ensemble, preventing hospitals with fundamentally different patient populations from degrading each other's models.

A technique using a domain discriminator with a gradient reversal layer to learn feature representations invariant to the client's domain. During training, the model simultaneously optimizes for task performance while penalizing features that reveal which institution the data came from. This mitigates feature distribution skew caused by different scanner manufacturers or demographic populations. The resulting representations form a domain-agnostic feature space where ensemble predictions become more consistent across sites.

An optimization framework that learns data representations which elicit the same optimal classifier across all training clients. Unlike standard empirical risk minimization, FIRM seeks causal relationships robust to spurious correlations that may exist in individual hospital datasets. The objective penalizes feature-classifier combinations where the optimal decision boundary varies across environments, forcing the model to discover genuinely predictive features rather than site-specific artifacts.

An approach that trains a model initialization across clients such that it can rapidly adapt to a new client's local data distribution with only a few gradient steps. Algorithms like Reptile and MAML variants find parameters that sit in a region of the loss landscape where fine-tuning is fast and effective. This complements ensemble methods by providing strong base models that can be personalized per site, combining the robustness of ensembling with the adaptability of meta-learning for extreme non-IID scenarios.