Federated Model Distillation

Instead of sharing model weights, clients share soft labels (logits) on a public, unlabeled dataset. The server averages these logits to create a consensus teacher ensemble, which distills knowledge into a global student model. This decouples aggregation from model architecture, allowing clients to use heterogeneous model topologies while still contributing to a unified global model.

Distillation dramatically reduces communication overhead compared to weight transfer. Key benefits include:

• Logit vectors are orders of magnitude smaller than model parameter tensors
• Communication cost scales with the public dataset size, not model size
• Enables participation from resource-constrained edge devices that cannot transmit full model weights
• Typical compression ratios of 100x to 1000x over Federated Averaging

A defining advantage of distillation is architecture-agnostic aggregation. Clients can train models with different:

• Neural network architectures (CNNs, transformers, MLPs)
• Layer counts and widths
• Optimization algorithms and hyperparameters This is critical in healthcare, where institutions may have varying computational resources and legacy model investments.

Distillation provides an additional privacy layer beyond federated learning. Clients never expose raw model parameters, only aggregated class predictions on public data. This mitigates:

• Model inversion attacks that reconstruct training data from gradients
• Membership inference attacks that determine if a record was in the training set
• Architectural leakage that reveals proprietary model design choices Combined with differential privacy noise on logits, strong formal privacy guarantees are achievable.

The student model is trained using specialized loss functions that transfer dark knowledge from the teacher ensemble:

• Kullback-Leibler divergence between softened teacher and student logits
• Temperature scaling (T > 1) to reveal inter-class relationships learned by teachers
• Combined with standard cross-entropy loss on any available labeled data
• Optional attention transfer to align intermediate feature representations

A key operational requirement is access to an unlabeled public proxy dataset that approximates the global data distribution. Considerations include:

• Dataset must be representative enough to capture teacher knowledge
• Can be sourced from open medical datasets or synthetically generated
• Domain gap between public and private data reduces distillation quality
• Federated data-free distillation variants eliminate this requirement using generative models

The foundational technique where a compact student model is trained to mimic the output distribution of a larger, more complex teacher model. In centralized settings, the student minimizes the divergence between its softened logits and the teacher's. This concept is extended to federated settings by treating the ensemble of local client models as the teacher collective, transferring their aggregated knowledge via predictions on a public, unlabeled dataset rather than sharing proprietary model parameters.

A variant where multiple teacher models are combined into a single student. In Federated Model Distillation, each client acts as an independent teacher. The server aggregates their logit outputs on a public dataset to form a consensus prediction, which the student then learns from. This approach naturally handles model heterogeneity—clients can have different architectures, capacities, and training objectives—since only output scores are communicated, not internal weights.

The process of combining the class score vectors from multiple client models into a single, knowledge-dense training signal for the student. Common strategies include:

• Arithmetic averaging of logits across clients
• Weighted averaging based on client performance or dataset size
• Majority voting on hard predictions
• Temperature scaling to soften probability distributions before aggregation This step is the communication bottleneck, transmitting only low-dimensional vectors instead of full model weights.

An unlabeled or labeled dataset accessible to all participants that serves as the transfer medium for knowledge. Clients generate predictions on this shared data, and the server uses these outputs to train the student. The quality and domain relevance of this dataset critically impacts distillation fidelity. Options include:

• Publicly available datasets (e.g., ImageNet, Wikipedia)
• Synthetic data generated by a separate model
• A held-out portion of unlabeled data agreed upon by the consortium

A primary motivation for federated distillation over weight-sharing approaches like FedAvg. Instead of transmitting millions of model parameters per round, clients send only logit vectors whose size equals the number of output classes. For a 1,000-class problem, this reduces communication from gigabytes to kilobytes per client per round. This makes distillation viable for cross-device federated learning where clients operate on bandwidth-constrained mobile or edge hardware.

Unlike weight-averaging methods that require all clients to share an identical model architecture, federated distillation imposes no architectural constraints. Clients can train CNNs, transformers, or even non-neural models like gradient-boosted trees. This enables participation from institutions with vastly different computational resources and legacy model investments, making it ideal for cross-silo healthcare networks where standardization is impractical.