Federated Distillation

Instead of transmitting high-dimensional model weights or gradients, clients share only the soft predictions (logits) generated on a public, unlabeled dataset. This decouples the communication cost from the model size, making it ideal for large neural networks where weight transfer would be prohibitive. The logit vector is typically much smaller than the parameter count, reducing bandwidth consumption by orders of magnitude.

Unlike Federated Averaging (FedAvg), which requires all clients to share an identical model architecture, federated distillation allows each client to maintain a bespoke local model tailored to its hardware constraints. A sensor node might use a compact CNN while a server uses a large transformer. The global model only needs to agree on the output logit dimension, enabling true architectural heterogeneity across the federated population.

The central server aggregates the received logits from all clients and uses them as soft targets to train the global model via knowledge distillation. This process transfers the dark knowledge embedded in the clients' output distributions—capturing inter-class similarities and uncertainty—rather than just hard label predictions. The global model learns a richer representation that generalizes better than simple weight averaging.

By transmitting only model outputs on a public reference dataset instead of raw model weights or gradients, federated distillation provides a natural privacy barrier. Logits reveal significantly less information about private training data than full model updates. This makes model inversion attacks substantially harder to execute, as the adversary cannot directly inspect the parameter space where sensitive data features might be encoded.

Federated distillation demonstrates superior resilience to statistical heterogeneity across client data distributions. Because the aggregation operates in the output space rather than the parameter space, the global model avoids the weight divergence problem that plagues FedAvg under non-IID conditions. The distillation process naturally reconciles conflicting label distributions by learning a consensus output manifold from diverse client logits.

A key operational requirement is the availability of a public, unlabeled surrogate dataset that captures the input distribution of the problem domain. This dataset serves as the communication medium—clients generate logits on it, and the server uses it for distillation. The quality and coverage of this public dataset directly impact the effectiveness of knowledge transfer. In practice, this can be sourced from open repositories or synthetically generated.

Unlike Federated Averaging (FedAvg) which requires identical model architectures, federated distillation allows each client to train a bespoke neural network tailored to its local hardware capabilities. A powerful server-side model (teacher) learns from the collective knowledge of diverse client models (students) by matching their soft label predictions on a public, unlabeled dataset. This enables collaboration between a 5G base station running a large transformer and an IoT sensor running a TinyML model, as they only need to agree on the output dimension, not the internal architecture.

In Over-the-Air Computation (AirComp) scenarios, transmitting full model weights is prohibitive. Federated distillation reduces communication payloads by orders of magnitude by exchanging only logit vectors for a public dataset. For a classification task with 1,000 classes, a client transmits a 1,000-element float vector per public sample instead of millions of weight parameters. This makes collaborative learning feasible over low-bandwidth links like LoRaWAN or tactical mesh networks, where the distillation payload is a fixed, small size regardless of the client model's complexity.

Federated distillation provides a natural privacy barrier because raw data and model weights never leave the client device. Only aggregated soft labels on a public, non-sensitive dataset are shared. This aligns with Differential Privacy principles, as the public dataset acts as an information bottleneck, limiting what can be leaked about private training samples. Unlike Model Inversion Attacks that can reconstruct training data from shared gradients in FedAvg, the logit vectors in distillation reveal only the model's consensus on public data, significantly reducing the attack surface for membership inference.

Statistical Heterogeneity causes severe model divergence in FedAvg when client data is Non-IID. Federated distillation mitigates this by aggregating knowledge at the output level rather than the parameter level. The server's global model is trained to mimic the ensemble of client predictions on a public dataset, effectively learning a consensus that smooths out local biases. This is analogous to ensemble distillation, where the collective wisdom of diverse teachers is more robust than any single model, making it particularly effective for Cross-Device Federated Learning with millions of statistically diverse users.

Combining federated distillation with Secure Aggregation or Homomorphic Encryption enables a powerful privacy paradigm. Clients can encrypt their logit vectors before transmission, and the server can perform the distillation loss computation directly on encrypted values. This ensures that even the aggregated soft labels remain confidential. In a Cross-Silo Federated Learning setting, competing telecom operators can collaboratively train a global Automatic Modulation Classification model without revealing their local model architectures, data distributions, or even the consensus predictions to each other, satisfying strict regulatory requirements.

Traditional synchronous Federated Averaging is bottlenecked by the slowest client, a problem known as the straggler effect. Federated distillation naturally supports Asynchronous Federated Learning because the server can update its global model whenever any client submits logits, without waiting for a full cohort. The server maintains a dynamic memory of recent client predictions and continuously distills knowledge. This is critical for Cross-Device deployments where mobile devices have intermittent connectivity and unpredictable availability, ensuring the global model improves monotonically without blocking on unresponsive nodes.