Federated Distillation

The central server aggregates soft label predictions from diverse client models on an unlabeled public reference dataset to train a single, high-performing global model. This process distills the collective knowledge of the federation without requiring any client to expose its private data or model architecture.

• Mechanism: Clients upload logit vectors, not weight matrices
• Benefit: The global model learns from the consensus of specialized local experts
• Medical context: A central diagnostic model learns to match the diagnostic accuracy of multiple specialized hospital models without ever seeing a single patient scan

Unlike weight-averaging methods such as FedAvg, federated distillation imposes no requirement that client models share identical architectures. Each hospital can train its own bespoke neural network—a Vision Transformer at one site, a CNN at another—tailored to its specific computational resources and data characteristics.

• Decoupled design: Clients are free to innovate on model architecture independently
• Hardware flexibility: A well-resourced academic hospital can deploy a large model while a rural clinic uses a lightweight mobile-friendly architecture
• Legacy compatibility: Existing institutional models can join the federation without modification

In co-distillation variants, clients not only contribute to a global model but also receive distilled knowledge back to improve their own local models. This creates a bidirectional knowledge transfer loop where every participant continuously benefits from the collective intelligence of the network.

• Process: Client A's model is trained to mimic the ensemble predictions of all other clients on the public dataset
• Outcome: Each local model gains exposure to patterns present in other hospitals' data distributions without direct data access
• Use case: A small hospital's model learns to recognize rare pathologies it has never encountered locally by distilling knowledge from larger institutions' diagnostic outputs

Transmitting model outputs rather than gradients or weights provides an inherent layer of privacy protection. Soft labels are significantly more difficult to exploit for model inversion attacks compared to raw gradient updates. When combined with differential privacy noise injection on the transmitted logits, the privacy guarantees are further strengthened.

• Reduced attack surface: Logits contain far less information about individual training samples than gradients
• Compatible with DP: Adding calibrated Gaussian noise to soft labels before transmission provides formal privacy bounds
• Regulatory alignment: This approach aligns with HIPAA and GDPR requirements for minimizing data exposure in multi-institutional collaborations

The choice of the public reference dataset is the critical design decision in federated distillation. This unlabeled dataset must be representative enough to elicit meaningful soft label distributions from all client models. In medical imaging, this often involves curating a diverse set of scans from publicly available repositories or generating synthetic images.

• Requirements: Dataset must span the input distribution space of all participating clients
• Sources: Public repositories like The Cancer Imaging Archive (TCIA), or synthetically generated medical images
• Risk: A poorly chosen reference set leads to distillation on out-of-distribution samples, degrading global model quality

A single communication round in federated distillation transmits only the output logits for each sample in the reference dataset, rather than millions of weight parameters. This makes the approach viable over constrained hospital network links and enables participation from edge devices with limited upstream bandwidth.

• Bandwidth profile: Communication cost scales with the size of the reference dataset × number of classes, not model size
• Comparison: A ResNet-50 has ~25 million parameters; transmitting logits for 10,000 images with 100 classes requires only 4MB per round
• Practical impact: Enables real-world cross-hospital training without requiring dedicated high-speed research network infrastructure

The foundational algorithm that exchanges model weights rather than outputs. In FedAvg, clients train locally and send updated weight matrices to a server for element-wise averaging. This contrasts sharply with distillation, where only soft labels on a public dataset are shared. FedAvg requires homogeneous model architectures across all clients, while distillation supports heterogeneous models—a critical advantage when hospitals use different neural network backbones.

The primary challenge in medical federated learning. Each hospital's imaging data reflects unique patient demographics, scanner vendors, and acquisition protocols. This statistical heterogeneity causes client drift in weight-based methods. Federated distillation partially mitigates this by sharing class-conditional soft labels rather than gradients, which can be more robust to label distribution skew. However, distillation still suffers when the public reference dataset poorly represents private data distributions.

A mathematical framework providing provable privacy guarantees through calibrated noise injection. In federated distillation, DP can be applied to the shared soft label vectors before transmission, bounding the information leakage about any single training sample. The privacy budget (epsilon) quantifies the trade-off: lower epsilon means stronger privacy but noisier, less useful distilled knowledge. Combining DP with distillation often yields better utility-privacy trade-offs than DP with weight sharing.

A cryptographic protocol ensuring the server can only compute the aggregated sum of client updates, never inspecting individual contributions. While originally designed for weight averaging, SecAgg can protect soft label histograms in distillation. Each client encrypts its output predictions, the server performs homomorphic addition, and only the aggregated soft labels are revealed. This prevents the server from reverse-engineering private data patterns from any single institution's predictions.

Techniques like quantization and sparsification that reduce the bit-size of transmitted updates. In weight-based FL, gradients are compressed to 8-bit or top-k sparse vectors. Federated distillation achieves inherent compression because soft label vectors are typically orders of magnitude smaller than weight matrices—a ResNet-50 has ~25 million parameters, while its output logits for 1,000 classes are just 1,000 floats. This makes distillation ideal for low-bandwidth hospital networks.

A privacy technique where the neural network is partitioned between client and server. The client computes up to a 'cut layer' and sends intermediate activations (smashed data), never raw images or gradients. Federated distillation complements split learning by using the server-side portion to generate soft labels for the public dataset, enabling hybrid architectures where the server never sees raw patient data but can still aggregate knowledge across institutions.