Federated Distillation

Instead of sharing gradients or weights, each client generates prediction vectors (logits) on a public reference dataset. These soft labels capture the client's learned decision boundaries and class similarities. The central server aggregates these predictions to train a global model, effectively distilling the collective intelligence without exposing private model architectures or local data distributions.

A core advantage over standard federated averaging: clients can use completely different model architectures. One client might use a ResNet while another deploys a lightweight MobileNet. Because only predictions on a public dataset are exchanged—not model parameters—the system is agnostic to each client's internal design, enabling participation from devices with vastly different compute capabilities.

Traditional federated learning is vulnerable to gradient inversion attacks, where shared parameter updates are mathematically reversed to reconstruct private training samples. Federated distillation eliminates this attack vector entirely by never transmitting gradients. The information bottleneck of sharing only output predictions provides a natural, robust defense against model inversion and membership inference.

The technique requires an unlabeled, task-relevant public dataset that all clients and the server can access. This dataset acts as the transfer medium for knowledge. The quality and distribution of this public data critically impact performance: it must be sufficiently representative of the task domain. In practice, this can be sourced from open repositories, synthetic data generation, or a held-out non-sensitive dataset provided by the coordinating party.

In the co-distillation paradigm, there is no central server. Clients exchange model predictions directly in a peer-to-peer fashion, each using the received soft labels to regularize or train their own local model. This decentralized approach eliminates the single point of failure and trust concentration, making it suitable for consortiums where no single entity should control the global model.

By transmitting only logit vectors for a fixed public dataset instead of full model weights, federated distillation dramatically reduces bandwidth requirements. A typical model update in FedAvg might be hundreds of megabytes, while a batch of soft labels is measured in kilobytes. This makes the technique particularly viable for edge devices, mobile networks, and cross-silo settings with constrained or expensive connectivity.

A symmetric variant where all participating models mutually learn from each other's predictions on a public dataset. Unlike standard federated distillation, there is no central student model; every client acts as both teacher and student. Mutual knowledge transfer reduces the single point of failure and allows for peer-to-peer model improvement without sharing proprietary architectures. The consensus is built through aggregated soft labels rather than parameter averaging.

A technique where the central server aggregates predictions from multiple teacher models to create a robust consensus label for the public dataset. The student model is trained to mimic this ensemble output, effectively compressing the collective intelligence of the distributed network. This approach provides inherent Byzantine resilience, as outlier predictions from malicious nodes are statistically diluted by the majority vote before distillation occurs.

Eliminates the dependency on a pre-existing public dataset by using a generative model to synthesize realistic proxy samples. The server or clients train a generator to produce inputs that maximize disagreement between teacher models, then distill the teachers' responses on these synthetic samples. This is critical when no suitable unlabeled proxy data exists, though it introduces risks of model inversion if the generator memorizes private patterns.

Applies formal privacy guarantees to the knowledge transfer process by adding calibrated noise to the aggregated predictions or the student model's gradients. The privacy budget (ε) quantifies the maximum information leakage. Key implementations include:

• Perturbing the soft-label histograms before transmission
• Applying DP-SGD during the student's training phase
• Using the PATE (Private Aggregation of Teacher Ensembles) framework to add noise to the voting mechanism

The foundational alternative to distillation-based federated learning. Clients train local models and send weight updates to the server, which computes a weighted average to update the global model. While efficient, FedAvg requires model homogeneity—all clients must share the same architecture. Federated distillation overcomes this limitation by communicating predictions instead of parameters, enabling heterogeneous model architectures across the network.

A primary threat vector that federated distillation is designed to mitigate. In standard federated learning, shared gradients can be inverted to reconstruct private training samples. Distillation replaces gradient sharing with prediction sharing on public data, severing the direct mathematical link between the update and the private input. However, distillation introduces its own risk: model inversion through repeated querying of the teacher's predictions.