Federated Distillation

Instead of sharing raw data or model weights, clients in Federated Distillation exchange distilled knowledge, typically in the form of soft labels (probability distributions) or compact embeddings. This significantly reduces the risk of data leakage through model inversion or membership inference attacks. For example, a client might only upload the teacher model's predictions on its local data, not the data itself, to a central server for aggregation.

A primary advantage over standard Federated Learning is the drastic reduction in communication overhead. Transmitting soft labels or small embeddings is far more bandwidth-efficient than exchanging millions of model parameters (gradients or weights). This is critical for training on edge devices with limited and expensive connectivity. The payload size is proportional to the number of data samples and output classes, not the model's parameter count.

Federated Distillation naturally accommodates statistical heterogeneity (non-IID data) and system heterogeneity (varying device capabilities).

• Model Heterogeneity: Clients can train different model architectures locally, as knowledge is transferred via outputs/embeddings, not gradient alignment.
• Data Heterogeneity: The distillation process, using softened labels, is more robust to label distribution skew across clients compared to methods requiring gradient averaging.

The process typically operates in synchronized rounds:

1. Local Distillation Phase: Each client uses its local data to train a student model, supervised by a combination of true labels and soft targets from a local or global teacher.
2. Knowledge Aggregation Phase: Clients send their locally generated soft labels or embeddings to a central server. The server aggregates this knowledge (e.g., by averaging) to update a global teacher model or to create an improved set of soft targets, which are then broadcast back to clients for the next round.

Unlike classic Federated Learning, there is often no single, monolithic global model whose parameters are averaged. The central server may maintain a lightweight global teacher model or simply aggregate soft knowledge. The primary output is the improvement of each client's local model through access to the ensemble's distilled knowledge. This is analogous to ensemble distillation, where the collective intelligence of all clients is compressed into each local model.

Federated Distillation is ideal for scenarios with strict privacy, bandwidth constraints, and heterogeneous hardware.

• Healthcare (Healthcare Federated Learning): Hospitals collaboratively train diagnostic models without sharing patient records.
• Mobile Keyboard Prediction: Improving next-word prediction across millions of phones without uploading typing data.
• Industrial IoT: Factories with different sensor setups and legacy equipment improving predictive maintenance models.
• Autonomous Vehicles: Fleets of vehicles learning to handle rare edge cases without transmitting sensitive video footage.

The core model compression technique where a compact student model is trained to mimic the behavior of a larger, more accurate teacher model. The student learns from the teacher's soft targets (probability distributions), which contain dark knowledge about inter-class relationships.

• Core Mechanism: Transfer of 'knowledge' via softened outputs or intermediate representations.
• Objective: Create a smaller, faster model that retains much of the teacher's performance.
• Application in FD: Clients use local models as teachers to produce soft labels for a global student model.

A collaborative learning technique where a model's architecture is vertically split between clients and a server. Each client holds the initial layers, processes its raw data, and sends the resulting intermediate activations or embeddings to the server, which completes the forward pass and computes the loss.

• Comparison to FD: Both avoid sharing raw data. Split Learning shares intermediate features; FD shares distilled outputs (soft labels).
• Privacy Aspect: Exposed embeddings may still leak information, whereas soft labels are a higher-level abstraction.
• Use Case: Useful when clients have limited compute but can handle initial layers.

A federated learning configuration involving a small number of reliable, resource-rich institutional clients (e.g., hospitals, banks) with large datasets. This contrasts with cross-device FL involving millions of unstable mobile devices.

• FD Context: Federated Distillation is often applied in cross-silo settings (e.g., healthcare, finance) where models are large, and communication efficiency is critical.
• Characteristics: Fewer clients (2-100), higher participation rate, more stable network connections.
• Motivation: Collaborate on sensitive data across organizational boundaries without a central data repository.

A distillation paradigm where the student model is trained without access to any real training data. Instead, a generator creates synthetic samples, often designed to maximize the teacher model's response, and the student learns from the teacher's outputs on these synthetic inputs.

• Connection to FD: In FD, the central server often lacks the original training data. The knowledge (soft labels) shared by clients acts as a data-free supervisory signal for the global model.
• Core Challenge: Generating informative synthetic samples that effectively transfer the teacher's knowledge.
• Technique: Often uses Generative Adversarial Networks (GANs) or gradient-based sample generation.