Federated Distillation

Unlike Federated Averaging (FedAvg) , which requires all clients to share an identical model architecture, federated distillation transmits only the soft labels (logits) produced by each local model on an unlabeled public dataset.

• A hospital using a ResNet-50 for chest X-ray analysis can co-train with a clinic running a lightweight MobileNetV3 on its edge devices.
• The server aggregates these predictions via ensemble distillation, producing a consensus output that trains a global student model without ever seeing a single model weight.
• This is critical for medical IoT ecosystems where devices range from powerful GPU servers to microcontroller-based wearables.

By design, federated distillation never transmits raw model parameters or gradients, significantly reducing the attack surface for model inversion and membership inference attacks.

• Local models only share their prediction vectors on a non-sensitive, publicly available dataset—often called the distillation dataset.
• This dataset can be synthetically generated or drawn from openly licensed medical literature images, ensuring no patient data is ever centralized or exposed.
• The technique provides a natural layer of differential privacy, as the shared logits represent aggregated model behavior rather than individual training samples.

Exchanging logits instead of full model weights dramatically reduces the bandwidth overhead of federated training rounds.

• A typical ResNet-50 contains over 23 million parameters, requiring ~90 MB per upload. In contrast, its output logits for a batch of 1,000 samples on a 10-class diagnostic task consume only ~40 KB.
• This three-order-of-magnitude reduction in payload size makes federated distillation ideal for rural clinics connected via low-bandwidth satellite links or 4G mobile networks.
• The technique also supports asynchronous updates, where slow or intermittently connected devices can contribute predictions on their own schedule without stalling the global model.

Federated distillation naturally extends to scenarios where different institutions are solving related but distinct clinical tasks on different label spaces.

• A cardiology department may classify 12-lead ECG rhythms, while a neurology unit analyzes EEG seizure patterns. Both can distill their domain-specific knowledge into a shared, multi-head model.
• The public dataset acts as a semantic bridge, allowing the global model to learn a unified feature representation that generalizes across modalities.
• This capability supports federated transfer learning, where knowledge distilled from one specialty accelerates training for another with limited labeled data.

Clinical data across hospitals is notoriously non-independent and identically distributed (non-IID) , with varying patient demographics, equipment vendors, and annotation protocols. Federated distillation is inherently more robust to this statistical heterogeneity than weight-averaging methods.

• Instead of forcing incompatible weight updates into a single model, distillation aggregates soft knowledge that captures nuanced class relationships and decision boundaries.
• A model trained on a predominantly geriatric population will produce logits reflecting that bias, but the distillation process can learn to calibrate these outputs against other institutions' predictions.
• This leads to better generalization and reduced catastrophic forgetting of local data distributions.

Beyond the client-server paradigm, federated distillation supports fully decentralized co-distillation where medical devices learn directly from each other without any central aggregator.

• In a mesh network of wearable cardiac monitors, each device can broadcast its output predictions on a shared set of unlabeled heartbeat samples.
• Neighboring devices use these predictions as auxiliary training targets, refining their own models based on the collective intelligence of the local network.
• This architecture eliminates the single point of failure and the regulatory complexity of a central server, aligning with sovereign AI infrastructure requirements for data governance.