Clustered Federated Learning

CFL algorithms, such as Iterative Federated Clustering Algorithm (IFCA), analyze the cosine similarity of local model updates to partition clients. The core assumption is that clients with similar data distributions produce gradients that point in similar directions in the weight space. This allows the server to group hospitals with comparable patient demographics or disease prevalence without ever inspecting raw patient data. The process is typically iterative, alternating between training cluster-specific models and reassigning clients based on the loss achieved on each cluster's model.

Many CFL frameworks employ a bi-level optimization structure:

• Upper Level: The server estimates the underlying data distribution clusters and manages the assignment of clients to these groups.
• Lower Level: Each client solves a local empirical risk minimization problem specific to its assigned cluster's model. This decoupling allows the system to learn both the global cluster structure and personalized local models simultaneously, often using techniques like Expectation-Maximization (EM) to handle the latent cluster assignment variable.

A prominent approach, CFL by Hypothesis, identifies divergence in client update directions during standard Federated Averaging. When the global model reaches a stationary point where different groups of clients pull the model in opposing directions, the server recursively splits the client population into separate clusters. This is mathematically formalized using the cosine similarity of gradient updates; a bi-modal distribution of cosine similarities signals the need for a split. This method is particularly effective for label distribution skew where one hospital treats predominantly disease A and another treats disease B.

To reduce communication overhead, some CFL methods perform clustering in a single communication round. Clients train locally on their private data and send only their final model weights (or a low-dimensional embedding of them) to the server. The server then applies standard clustering algorithms like K-Means or DBSCAN directly on the received weight vectors. This approach assumes that models trained on similar data distributions will converge to similar regions in the weight loss landscape, a property exploited by model-zoo based federated learning.

Rigid cluster assignment can be suboptimal for clients with data that straddles multiple distributions. Soft clustering or fuzzy C-Means approaches assign each client a probability vector over all clusters. The final personalized model for a client is a weighted ensemble of all cluster-specific models, where the weights correspond to the client's cluster membership probabilities. This provides a smooth interpolation between purely global and purely local models, gracefully handling concept drift and mixed clinical presentations.

CFL introduces overhead for cluster assignment and model dissemination. Optimizations include:

• Compressed Gradient Clustering: Clients send sketched or quantized gradients for the clustering step, reducing uplink bandwidth.
• Cluster Head Selection: Only a representative subset of clients in each cluster participates in every round, with others synchronizing periodically.
• Dynamic Cluster Merging: When two cluster models become statistically indistinguishable, they are merged to reduce the total number of models the server must maintain and broadcast, preventing unnecessary fragmentation of the client population.