Federated Distribution Matching

The core mechanism of FDM is the explicit minimization of a statistical divergence metric—such as Kullback-Leibler (KL) divergence, Maximum Mean Discrepancy (MMD), or Wasserstein distance—between each client's local data distribution and a predefined global target distribution. Unlike standard Federated Averaging, which only aligns model weights, FDM directly operates on the data representation space to enforce distributional similarity. This is typically achieved by adding a divergence penalty term to the local loss function, forcing the model to learn features that are invariant to the domain shift between clients.

FDM directly addresses the non-IID problem in federated learning by tackling the root cause: statistical heterogeneity in local data distributions. When a model trained on one hospital's imaging equipment is applied to another's, the shift in pixel intensity distributions can cause catastrophic performance degradation. FDM mitigates this by aligning the latent feature distributions across clients, ensuring that the global model learns a domain-invariant representation. This is critical in healthcare, where scanners from different manufacturers produce systematically different image characteristics.

A practical implementation of FDM often involves the server maintaining a synthetic global distribution or a set of prototypical feature representations. Clients receive this target distribution and locally train to match it. The server can update this target by aggregating distributional statistics—not raw data—from clients, preserving privacy. Techniques include:

• Feature covariance alignment: Matching second-order statistics across clients.
• Prototype aggregation: Averaging class-specific feature centroids from each client.
• Generative replay: Using a server-side generative model to produce samples representing the global distribution.

FDM is a foundational enabler for Federated Transfer Learning (FTL) in scenarios with severe feature space misalignment. When source and target domains have different marginal distributions P(X), FDM aligns these distributions before or during the transfer process. This is distinct from standard FTL, which primarily addresses label space and feature space mismatches. By ensuring that the input distributions are statistically compatible, FDM allows knowledge transfer to occur without the confounding factor of domain shift, improving the generalization of transferred representations.

FDM introduces a distinct communication pattern compared to standard federated averaging. Instead of transmitting only model weights, clients may need to share distributional statistics such as feature means, covariance matrices, or kernel embeddings. This creates a trade-off:

• Increased per-round payload: Transmitting covariance matrices scales quadratically with feature dimension.
• Reduced total rounds: Faster convergence due to aligned distributions can decrease the total number of communication rounds.
• Compression techniques: Low-rank approximations and random Fourier features can reduce the overhead of transmitting distributional information.

Sharing distributional statistics rather than raw data provides a baseline level of privacy, but membership inference risks remain. An adversary could potentially reconstruct sensitive attributes from shared covariance matrices or feature prototypes. To harden FDM, it is often combined with formal privacy guarantees:

• Differential Privacy (DP): Adding calibrated noise to the shared distributional statistics before transmission.
• Secure Aggregation: Ensuring the server only sees the aggregated global distribution, not individual client contributions.
• Distributional Clipping: Bounding the sensitivity of the shared statistics to limit information leakage.