Federated Mutual Learning

Unlike traditional teacher-student distillation, FML establishes a peer-teaching network where every model acts as both teacher and student. Each client model trains on its local private data and a public dataset, then shares its logit predictions (soft labels) on the public dataset with peers. This mimics the information density of centralized training by allowing models to learn from the collective posterior distribution of the entire cohort, not just hard labels.

A cohort of K local models with identical architectures but different initializations trains simultaneously. Each model maintains its own trajectory through parameter space, preserving hypothesis diversity. During mutual learning steps, models exchange predictions on a shared unlabeled public dataset, using the Kullback-Leibler divergence between their own predictions and the cohort's aggregated predictions as an additional regularization term. This prevents premature consensus and maintains exploration.

FML achieves privacy by design through its prediction-only sharing protocol. Raw data, model weights, and gradients never leave the local institution. Only softmax outputs on a non-sensitive public dataset are exchanged. This provides a natural defense against gradient leakage attacks and model inversion, as the shared information is a highly compressed, task-specific representation that cannot be trivially decoded back to training samples.

FML inherently handles non-IID data distributions across clinical sites without requiring distribution alignment. Because each model maintains its own parameters and only shares predictions, divergent local optima do not force a compromised global consensus. Models with rare disease cases teach others about those distributions through their predictions, effectively performing implicit domain adaptation without explicit distribution matching or client clustering.

FML reduces communication overhead compared to weight-sharing federated learning. Instead of transmitting full model parameters (potentially millions of values), clients exchange only class probability vectors on a modestly sized public dataset. This is particularly advantageous in healthcare settings with limited bandwidth between rural clinics and central servers. The communication cost scales with O(K × |D_public| × C) where C is the number of classes, not model size.

During training, each model benefits from the collective intelligence of the cohort through mutual distillation. At inference time, any single model can be deployed independently, achieving accuracy approaching that of an ensemble without the computational overhead of running multiple models. This is critical for edge deployment on medical devices where inference latency and memory are constrained, yet diagnostic accuracy must remain high.