Federated Distillation

Clients exchange only the soft labels (logits or probability vectors) produced by their local models on a public or synthetically generated dataset, rather than sharing model parameters. This compresses the communication payload to the size of the output layer, which is typically orders of magnitude smaller than the model itself. The central server aggregates these predictions to train a global student model via knowledge distillation, where the soft targets reveal the dark knowledge of local models' decision boundaries.

Unlike weight-sharing methods such as Federated Averaging (FedAvg), federated distillation imposes no constraints on local model architecture. Each client can independently choose its neural network topology, layer count, or even model family. This enables heterogeneous system design where resource-constrained edge devices deploy compact models while well-resourced institutions use large foundation models, all contributing to the same global consensus through a unified logit interface.

In co-distillation, the server maintains a public dataset (which may be unlabeled or synthetically generated) and distributes it to all clients. Each client runs inference on this dataset and returns the resulting logit vectors. The server aggregates these soft labels—typically via arithmetic averaging—and uses them as targets to train a global student model. This student can then be redistributed to clients, creating a bidirectional knowledge loop without exposing private local data.

Federated distillation provides an inherent privacy barrier because raw data and model weights never leave the client. Only aggregated predictions on a public dataset are transmitted. This can be further hardened with differential privacy by adding calibrated noise to the shared logits before transmission. The distillation process also acts as a natural defense against model inversion attacks, as the attacker receives only output distributions rather than the parameter space of the local model.

The communication cost scales with O(C × |D_pub|) where C is the number of classes and |D_pub| is the size of the public dataset, rather than the millions or billions of parameters in the model. For a 10-class problem with 10,000 public samples, each client transmits only 100,000 floating-point values per round. This represents a compression ratio exceeding 99.9% compared to transmitting a ResNet-50, making federated distillation viable over bandwidth-constrained hospital networks.

The server aggregates logits from multiple clients to form an ensemble teacher. This ensemble captures the collective knowledge and uncertainty calibration of all participating models. The global student model is trained to mimic this ensemble's predictions, effectively distilling diverse local expertise into a single unified model. This approach is particularly robust to non-IID data distributions, as the ensemble naturally balances conflicting local patterns without the weight divergence issues that plague FedAvg.