Client Clustering

Clustering algorithms analyze local data distributions or model update similarities to group clients with compatible statistical properties. This prevents a single global model from being pulled in conflicting directions by clients with fundamentally different data generation processes. Common similarity metrics include cosine similarity between gradient updates, Earth Mover's Distance between feature distributions, and KL divergence between local label distributions. By ensuring each cluster trains on relatively homogeneous data, standard aggregation algorithms like FedAvg can operate effectively within each group.

A recursive clustering approach that starts with a single global model and progressively splits clients into subgroups when the consensus error exceeds a threshold. At each round, the server evaluates whether the current cluster's aggregated update adequately represents all members. If not, it partitions the cluster into two using spectral clustering on client gradients. This continues until each subgroup achieves stable convergence. The algorithm dynamically adapts to emergent data distribution shifts without requiring pre-specified cluster counts.

Unlike standard federated learning which maintains one global model, clustered approaches require the server to manage multiple concurrent global models—one per identified cluster. The server must track cluster membership, route model updates to the appropriate aggregation pipeline, and handle clients that may transition between clusters as their local data evolves. This architecture introduces additional state management complexity but enables the system to serve distinct population subgroups with specialized models optimized for their specific characteristics.

Client clustering introduces additional communication rounds for similarity computation and cluster assignment. The server may need to request model updates or distribution statistics from clients specifically for clustering purposes before performing aggregation. Techniques to mitigate this include:

• One-shot clustering based on initial local training
• Compressed gradient sketches for similarity comparison
• Asynchronous cluster updates that don't block training rounds
• Hierarchical aggregation where cluster heads perform intermediate merging

When a new client joins the federated network, the system must determine its cluster assignment without prior knowledge of its data distribution. Strategies include:

• Warm-start inference: Running a few local epochs and comparing the resulting update to existing cluster centroids
• Auxiliary metadata: Using available demographic or institutional descriptors to predict cluster membership
• Probationary period: Temporarily assigning the client to the largest cluster while monitoring its contribution for potential reassignment
• Fallback global model: Maintaining a general-purpose model for unclassifiable clients

Client clustering is conceptually related to clustered multi-task learning, where tasks are grouped based on their relatedness. In federated settings, each client represents a distinct task with its own data distribution. Clustering identifies groups of clients whose tasks are sufficiently similar to benefit from shared learning. This contrasts with pure personalization approaches like FedPer or Ditto, which give every client a unique model. Clustering occupies a middle ground—providing specialization without the full overhead of per-client model maintenance.

A recursive bi-partitioning framework that separates clients based on the cosine similarity of their local gradient updates. Rather than pre-defining the number of clusters, CFL iteratively splits the client population when the global model's convergence stagnates due to divergent local objectives. This creates a hierarchical tree of models, ensuring that clients within a leaf node share a congruent optimization landscape.

A multi-armed bandit-inspired approach that dynamically groups clients by testing statistical hypotheses on their local empirical risk. The server maintains a set of candidate clustering structures and selects the one that minimizes the maximum generalization gap across clients. This method is particularly effective when data distributions are non-stationary, as it continuously re-evaluates cluster assignments based on streaming model performance.

A structural alternative to explicit clustering that treats each client as a separate task within a regularized multi-task framework. By enforcing a low-rank or clustered structure on the task relationship matrix, this approach implicitly discovers groupings without requiring a hard partitioning step. Clients with similar data distributions naturally share statistical strength through the learned task covariance matrix.

An alternating minimization algorithm that jointly estimates cluster identities and model parameters. In each round, clients are assigned to the cluster whose model yields the lowest local empirical loss, after which standard federated aggregation is performed within each group. IFCA requires a pre-specified number of clusters and is robust to initialization, often converging to a stable partitioning within a few communication rounds.

A clustering paradigm that groups clients not by data similarity but by their worst-case loss profiles. The server constructs clusters such that a single model minimizes the maximum loss across all members of the group. This is critical in clinical settings where guaranteeing a minimum performance floor for every hospital in a cluster is more important than optimizing average accuracy.

A communication-efficient technique where clients perform local training and transmit only compressed model representations or data distribution sketches to the server. The server then clusters these embeddings centrally without iterative communication. This approach is ideal for cross-silo healthcare networks with strict bandwidth constraints or asynchronous participation patterns.