Federated Knowledge Distillation

Instead of transmitting gradients or weights, clients share soft labels (logits or probability distributions) generated on a public, unlabeled reference dataset. The server aggregates these soft predictions to train a global student model via knowledge distillation. This decouples the training process from the model architecture, as only the output layer dimensionality needs to match. Soft labels inherently carry richer information than hard labels, encoding the teacher's uncertainty and inter-class relationships, which improves generalization on non-IID data.

FKD is uniquely suited for environments where clients possess different model architectures, capacities, or compute budgets. Because only predictions are shared, a hospital running a lightweight CNN can collaborate with a research center using a large vision transformer. The global student model can be a completely different architecture from any client teacher. This architectural agnosticism is critical for real-world healthcare networks where legacy systems and diverse hardware coexist.

Transmitting soft labels on a public dataset is often orders of magnitude smaller than transmitting full model weights. For a classification task with C classes and a public dataset of N samples, the communication cost per round is O(N × C) rather than O(millions of parameters). This makes FKD viable for bandwidth-constrained environments like rural clinics or edge devices. Techniques like top-k logit sparsification and entropy-based sample selection further compress the payload.

By design, FKD never exposes raw model parameters or gradients, reducing the attack surface for model inversion and membership inference attacks. The shared soft labels are aggregated across clients, providing a natural form of anonymization. When combined with differential privacy—adding calibrated noise to the aggregated soft labels—formal privacy guarantees can be established. However, the selection of the public dataset is critical; it must not overlap with private training data to prevent label leakage.

FKD demonstrates strong resilience to label distribution skew and feature distribution skew across clients. Because the global student learns from aggregated soft labels rather than averaged parameters, it avoids the weight divergence problem that plagues Federated Averaging under non-IID conditions. The distillation process naturally smooths out conflicting client updates. Techniques like logit ensemble weighting and client confidence calibration further mitigate the impact of outlier clients with poor local data quality.

Advanced FKD protocols extend beyond a simple teacher-student hierarchy. In co-distillation, all clients act as both teachers and students, distilling knowledge from peers while training on local private data. Federated mutual distillation allows clients to collaboratively train without a central server by exchanging soft labels in a peer-to-peer topology. These variants eliminate the single point of failure and are well-suited for consortium-based medical research networks where no single institution acts as the central aggregator.

The fundamental challenge that Federated Knowledge Distillation is designed to solve. In clinical settings, data is rarely independent and identically distributed across hospitals. One site may specialize in rare diseases (label distribution skew), while another uses a different MRI scanner vendor (feature distribution skew). Knowledge distillation bypasses the need to average incompatible model weights by sharing only soft predictions on a public, unlabeled dataset.

The process of fine-tuning a globally distilled model on local client data. After clients collaboratively train a teacher ensemble via knowledge distillation, each institution can personalize the resulting student model on its own private data. This two-stage approach—global distillation followed by local adaptation—is highly effective for handling site-specific patient demographics without compromising privacy.

A paradigm where knowledge distillation excels when feature spaces or label spaces differ across clients. For example, a hospital with imaging data and another with genomic data cannot train a single shared model. Federated Knowledge Distillation allows them to use a public proxy dataset to transfer knowledge between their distinct, heterogeneous models by matching their output distributions.

A closely related strategy where multiple independently trained local models are combined via their predictions rather than their parameters. Federated Knowledge Distillation can be viewed as compressing a federated ensemble into a single, deployable student model. The ensemble's aggregated soft labels on a public dataset capture the collective knowledge of all clients, providing a richer training signal than hard labels.

A communication-efficient alternative where clients share compact class-representative vectors (prototypes) instead of full model updates or logit vectors. While knowledge distillation transmits soft predictions on a public dataset, prototype learning transmits averaged embeddings per class. Both methods naturally handle label distribution skew and reduce communication costs compared to transmitting full model weights.