Federated Ensemble Distillation

Each institution independently trains multiple diverse models on its private clinical data. This local ensemble captures a robust, high-dimensional representation of the site's unique patient distribution.

• Diversity Sources: Different initializations, architectures (CNN, Transformer), or hyperparameters.
• Output: A set of logit vectors representing the local 'dark knowledge' of class similarities.
• Privacy Guarantee: Raw patient records never leave the institution's secure perimeter.

The central server queries each local teacher ensemble using a public, unlabeled proxy dataset or synthetically generated samples. The teachers return their predictive probability distributions (soft labels), not raw data or gradients.

• Knowledge Carrier: The averaged logits from all sites form a rich, privacy-compliant training signal.
• Heterogeneity Handling: Averaging soft labels naturally smooths out inter-site statistical variance.
• Communication Efficiency: Only low-dimensional probability vectors are transmitted, not massive model weights.

A single, compact student model is trained centrally on the aggregated soft labels using a distillation loss function, typically Kullback-Leibler divergence. The student learns to mimic the consensus behavior of the entire distributed network.

• Temperature Scaling: A high-temperature softmax is applied to soften the probability distribution, revealing the subtle inter-class relationships learned by the teachers.
• Loss Function: L = α * KL(soft_labels || student_logits) + (1-α) * CE(hard_labels, student_logits)
• Result: A lightweight, deployable model that encapsulates the collective intelligence of all institutions.

The quality of the distilled student model is directly proportional to the diversity of the local teacher ensembles. Techniques are employed to ensure teachers make different errors, providing richer supervisory signal.

• Dropout as Diversity: Different dropout masks at inference time create an implicit ensemble from a single model.
• Adversarial Weight Perturbation: Adding controlled noise to local model weights before logit generation.
• Architectural Heterogeneity: Encouraging sites to train structurally distinct models (e.g., ResNet vs. DenseNet) to capture complementary feature representations.

To prevent the aggregated soft labels from leaking membership information about individual patients, differential privacy (DP) noise is injected into the teacher logits before transmission or during aggregation.

• Gaussian Mechanism: Calibrated Gaussian noise is added to the averaged logit vector, providing a formal (ε, δ)-DP guarantee.
• Privacy-Utility Trade-off: The privacy budget ε controls the noise magnitude; a lower ε provides stronger privacy but may degrade the student model's accuracy.
• PATE Framework: This architecture is a direct implementation of the Private Aggregation of Teacher Ensembles framework.

The central distillation process requires a transfer set of unlabeled data that is representative of the overall clinical domain. This data must not overlap with any institution's private training set.

• Sources: Publicly available medical datasets, synthetically generated patient records from a federated GAN, or a held-out validation pool from a non-participating institution.
• Domain Alignment: The proxy set's feature distribution must approximate the aggregate distribution of the private silos for effective knowledge transfer.
• Synthetic Preference: Using a federated synthetic data generator is preferred as it avoids any potential privacy entanglement with real patient data.

The foundational knowledge distillation paradigm where a cumbersome, high-capacity teacher ensemble—comprising multiple models trained independently at separate sites—generates soft labels. A compact student model is then trained centrally on an aggregation of these soft probabilistic outputs, learning the teachers' generalized decision boundaries and inter-class similarities rather than just hard labels. This architecture decouples model compression from raw data access.

The mathematical process of combining the output logits or probability distributions from disparate local teacher models into a unified training signal for the central student. Common aggregation strategies include:

• Averaging logits: Simple mean of output vectors.
• Weighted averaging: Weighting teacher contributions by local validation accuracy or dataset size.
• Majority voting: For hard-label distillation variants. This step is critical for resolving inter-teacher disagreements and transferring consensus knowledge.

Unlike standard federated averaging (FedAvg), which shares model gradients or weights—vulnerable to inversion attacks—ensemble distillation shares only model outputs on a public or synthetically generated dataset. This output-based communication provides a stronger differential privacy guarantee by default, as the raw patient data and internal model parameters never leave the local institution. It is a natural fit for HIPAA and GDPR compliance.

A key enabler of federated distillation is the use of an unlabeled proxy dataset—often publicly available or synthetically generated—to query the local teachers. Each teacher generates soft predictions on this shared dataset, which are then aggregated to train the student. This elegantly sidesteps the need for a central validation set with sensitive labels and eliminates the requirement for non-IID data alignment across silos.

The power of distillation stems from the diversity of the teacher ensemble. In a federated context, this diversity is naturally induced by the heterogeneous, non-IID data distributions across institutions. A hospital specializing in oncology and another in cardiology will produce teachers with complementary expertise. The student model learns a more robust and generalized hypothesis by mimicking this combined, diverse knowledge, often outperforming any single local model.

A critical architectural comparison:

• FedAvg: Exchanges model weights/gradients. Requires identical model architectures. High communication cost for large models. Vulnerable to gradient leakage.
• Ensemble Distillation: Exchanges output logits on a proxy dataset. Supports heterogeneous local architectures (teachers can be different model types). Communication cost depends on proxy dataset size, not model size. Inherently more private. Preferred when local compute is high but trust is low.