Knowledge Distillation Aggregation (FedKD)

Instead of averaging parameter vectors, FedKD aggregates knowledge by matching the soft labels (logit outputs) of local teacher models. The global student model is trained to minimize the Kullback-Leibler (KL) divergence between its softened output distribution and the aggregated outputs of the teacher ensemble. This operates on a model-agnostic level, allowing teachers and the student to have completely different architectures.

A primary advantage over Federated Averaging (FedAvg) is the ability to fuse models with non-identical architectures. Local clients can train CNNs, RNNs, or Transformers of varying widths and depths. The global server only requires access to their output logits on a public or synthetically generated dataset, abstracting away the internal parameter structure and enabling cross-modal fusion.

The server acts as a central distillation hub. It queries all local teacher models with an unlabeled public dataset to collect their predictive distributions. These distributions are aggregated (e.g., via simple averaging) to form a consensus teacher ensemble. The global student model is then trained via standard knowledge distillation to mimic this ensemble, effectively compressing distributed knowledge into a single deployable model.

FedKD enhances privacy by design because raw model parameters and gradients are never transmitted to the server. Only the model's output logits on a public reference dataset are shared. This breaks the direct link to private training data, providing a natural defense against gradient inversion and membership inference attacks, though it requires careful curation of the distillation dataset to prevent information leakage.

FedKD demonstrates strong resilience to statistical heterogeneity. Since the global student learns from the aggregated output behavior rather than conflicting parameter directions, it avoids the weight divergence problem common in FedAvg. The distillation process naturally smooths out local biases, converging to a solution that captures the consensus knowledge across diverse, non-IID clinical data silos.

While FedKD eliminates the need to transmit large model weights, it introduces a query-based communication cost. The server must send the public dataset to clients, and clients must return logit vectors for every sample. This is efficient for large models with small output spaces but can become a bottleneck for tasks with high-dimensional outputs. Techniques like logit quantization are often applied to mitigate this overhead.

The foundational weight-space aggregation algorithm that constructs a global model by computing an element-wise weighted average of locally trained model parameters. Unlike FedKD, which operates on logit outputs, FedAvg directly merges model weights, requiring all clients to share identical model architectures. This makes FedAvg incompatible with heterogeneous model deployments but highly efficient when architectural homogeneity is guaranteed.

A model fusion technique that treats locally trained models as an independent ensemble, combining their predictions via voting or averaging at inference time rather than distilling them into a single global model. FedKD improves upon this by compressing the ensemble's collective knowledge into a compact student model, reducing inference latency and eliminating the need to maintain multiple teacher models in production.

An aggregation framework using Moreau envelopes to decouple personalized model optimization from global model learning. While pFedMe focuses on parameter-space personalization through regularization, FedKD achieves personalization through soft label alignment, allowing local models to retain unique predictive characteristics without requiring complex bi-level optimization procedures.

A class of robust aggregation rules—including Krum and Trimmed Mean—designed to ensure convergence when malicious nodes submit corrupted updates. FedKD provides inherent Byzantine resilience because distillation operates on soft labels rather than raw gradients. Anomalous logit distributions from compromised clients are naturally attenuated during the KL divergence minimization process, as they diverge from the ensemble consensus.

An aggregation strategy that excludes batch normalization layers from global averaging, preserving local batch statistics to combat feature shift across sites. FedKD inherently avoids batch norm mismatch issues because it never transfers parameters—only soft label distributions. This makes FedKD particularly suitable for cross-institutional medical imaging where scanner-specific normalization varies dramatically.

An aggregation mechanism injecting calibrated Gaussian noise into model updates to provide formal privacy guarantees. When combined with FedKD, differential privacy can be applied to the distilled logit outputs rather than high-dimensional weight vectors, achieving tighter privacy-utility tradeoffs due to the lower sensitivity of soft labels compared to raw gradients in weight-space aggregation.