Federated Knowledge Distillation

Instead of sharing sensitive model weights or gradients, clients share only the soft labels (logit vectors) generated by a teacher model on a public, unlabeled dataset. These logits encode rich, dark knowledge about inter-class similarities that one-hot labels miss.

• Mechanism: A global teacher model processes a public dataset and distributes its output probabilities to local student models.
• Privacy Benefit: Raw patient data and model parameters never leave the local institution.
• Example: A teacher model trained on chest X-rays across hospitals outputs a probability distribution [0.7 pneumonia, 0.2 edema, 0.1 normal] for a public image, teaching students about diagnostic nuance.

FKD enables institutions with vastly different computational resources to collaborate without forcing a common model architecture. Each client can train a bespoke student model tailored to its hardware constraints.

• Architectural Freedom: One hospital may use a lightweight MobileNet for edge deployment, while another trains a large Vision Transformer.
• Knowledge Alignment: All students learn from the same teacher's output distribution, ensuring consistent diagnostic logic despite architectural differences.
• Clinical Relevance: A rural clinic with limited GPU resources can distill knowledge from a consortium's powerful teacher model into a compact, on-premise student model.

The entire distillation process relies on a public, unlabeled proxy dataset that acts as a communication medium. This dataset must be representative enough to capture the teacher's decision boundaries without containing any private patient information.

• Selection Criteria: The proxy dataset should cover the input distribution of the target task (e.g., diverse chest X-rays from public repositories like CheXpert or MIMIC-CXR).
• No Label Requirement: The proxy data does not need ground-truth labels; the teacher's predictions serve as the training signal.
• Risk: A poorly chosen proxy dataset that does not match the federated data distribution can lead to distillation collapse, where the student learns a degraded approximation.

FKD dramatically reduces communication overhead compared to traditional Federated Averaging (FedAvg). Instead of transmitting millions of model parameters per round, only logit vectors for the proxy dataset are exchanged.

• Bandwidth Reduction: Transmitting logits for a few thousand public samples requires orders of magnitude less bandwidth than sharing full model weights.
• Asynchronous Operation: Clients can download the latest teacher logits and perform local distillation at their own pace, without synchronous coordination.
• Metric: For a ResNet-50 model (25M parameters), FedAvg transmits ~100MB per round, while FKD transmits logits for 10,000 samples at ~40KB.

The global teacher model can be an ensemble of multiple local models, whose logits are averaged on the server before distribution. This ensemble knowledge distillation provides superior generalization and robustness.

• Uncertainty Reduction: Averaging predictions from diverse local models reduces epistemic uncertainty and smooths out individual biases.
• Byzantine Resilience: The impact of a malicious or poorly trained client is diluted when its logits are averaged with many others.
• Clinical Benefit: An ensemble teacher trained across cardiology, radiology, and pulmonology departments produces more reliable diagnostic signals than any single institution's model.

While logit exchange is inherently more private than weight sharing, the teacher's predictions can still leak information about the private training data. FKD can be hardened by applying differential privacy (DP) to the aggregated logits.

• DP Mechanism: Gaussian noise is added to the averaged teacher logits before distribution, bounding the privacy loss with a quantifiable epsilon budget.
• Privacy-Utility Trade-off: Higher noise levels provide stronger privacy guarantees but degrade the quality of the distilled knowledge.
• Auditability: The privacy budget consumed per distillation round can be tracked and limited, providing compliance officers with a verifiable privacy posture.

A core challenge in federated learning that distillation helps mitigate. Client drift occurs when local models trained on non-IID data diverge from the optimal global solution. Knowledge distillation addresses this by using a consistent teacher signal from a public dataset rather than averaging divergent weight spaces. This provides a more stable optimization target, reducing the variance introduced by heterogeneous local data distributions.

A privacy framework often layered with federated distillation. While distillation avoids direct weight transfer, the teacher's soft predictions can still leak information. Applying differential privacy to the teacher's outputs—by adding calibrated noise to the logits—provides formal privacy guarantees. The privacy budget (epsilon) must be tracked across distillation rounds to bound total information leakage from the public dataset queries.

The broader paradigm that federated knowledge distillation enables. Rather than forcing all clients to use a single global model, distillation allows each client to train a custom student architecture optimized for local data characteristics and hardware constraints. This supports heterogeneous deployment scenarios where a powerful server-side teacher guides lightweight on-device students, achieving personalization without compromising the privacy benefits of federated training.

A direct application of knowledge distillation in federated settings. The global teacher model is typically large and computationally expensive, while student models are compressed versions suitable for edge deployment. Distillation transfers the teacher's knowledge to smaller architectures like MobileNet or distilled transformers without requiring clients to run the full model. This enables deployment on medical edge devices with limited compute and memory budgets.

A complementary cryptographic protocol often used alongside distillation. While distillation uses public data for knowledge transfer, SecAgg protects the model update aggregation process when weight sharing is still required for the teacher model. The teacher itself may be trained via federated averaging with SecAgg, then its predictions are used for distillation. This creates a hybrid privacy architecture combining cryptographic and distillation-based protections.