Personalized Model Aggregation

Instead of a uniform average, the server calculates a weighted average of client model updates where the weights are specific to the target client. Weights are often based on client similarity metrics, such as cosine similarity between model updates or data distribution proxies. This ensures a client's aggregated model is influenced more by clients with similar data patterns.

• Example: Client A receives an update heavily weighted from Clients B and C, whose model gradients are most similar to A's, while Client D receives a different blend.

The server maintains a memory or buffer of past model updates from each client. The aggregation for a client in the current round can incorporate its own historical updates or those of others. This allows the server to compensate for client drift—where local training diverges from the global objective—by anchoring updates to a client's historical trajectory or by identifying consistent patterns.

• Mechanism: Techniques may use an exponential moving average of a client's past parameters to stabilize the personalized aggregation point.

The aggregation rule is explicitly designed to optimize for local performance on each client's data distribution, not just global average accuracy. The server's aggregation function often includes terms that minimize the expected loss on each client's local distribution. This contrasts with FedAvg, which implicitly assumes homogeneous (IID) data and aims for a single global optimum.

• Formal Goal: Find a set of personalized models {θ_i} that perform well on their respective local distributions D_i.

Personalized aggregation is closely related to clustered federated learning. In clustered FL, clients are partitioned into groups with similar data, and a separate global model is trained per cluster. Personalized aggregation can be seen as a soft, dynamic clustering performed at aggregation time, where the "cluster" (weighting) is recomputed per client per round, allowing for more granular personalization than fixed, hard clusters.

Specific algorithms implement personalized aggregation rules:

• FedFomo: Clients receive personalized updates by selecting and weighting other clients' models based on validation performance on their local data.
• Per-FedAvg: Uses a meta-learning formulation; the server aggregates to find a global model initialization that is good for performing a few steps of personalized fine-tuning on any client.
• pFedMe: Separates the personalized model (local) from a global model, using a Moreau envelope during aggregation to balance personalization and global consensus.

This technique introduces computational and communication overhead compared to FedAvg.

• Server Compute: The server must run a distinct aggregation routine for each client, not just one global average.
• Client-Server Communication: May require clients to send additional metadata (e.g., similarity vectors) to inform the weighting process.
• Storage: The server must store per-client state (historical updates). The benefit is superior personalized accuracy in heterogeneous environments, justifying the overhead for critical use cases.

Personalized Federated Learning (PFL) is a decentralized machine learning paradigm where a global model is collaboratively trained across multiple clients, but the final deployed model is tailored or adapted to the unique data distribution of each individual client. The core challenge is balancing the benefits of collaborative learning with the need for local relevance.

• Goal: Produce a set of models {θ₁, θ₂, ..., θₙ} for N clients, where each θᵢ performs well on client i's local data distribution Dᵢ.
• Contrast with Standard FL: Standard FL aims for a single global model θᴳ that generalizes across all clients, which often fails under significant data heterogeneity (non-IID data).

Personalized Federated Averaging (pFedAvg) is a foundational PFL algorithm that modifies the standard Federated Averaging (FedAvg) process. Instead of deploying the aggregated global model directly, each client performs additional local fine-tuning on the received global model using its private data before inference.

• Mechanism: Server aggregates updates via FedAvg → broadcasts global model θᴳ → each client i performs k steps of SGD on θᴳ with its local data to produce personalized model θᵢ.
• Key Insight: The global model serves as a strong, communication-efficient initialization point for rapid local adaptation.

FedPer and FedRep are canonical PFL algorithms based on layer-wise personalization. They architecturally decouple the model into shared base layers and personalized head layers.

• FedPer: Keeps the base layers (feature extractor) global and federated. The personalized layers (typically the final classification head) are local, trained solely on client data, and never aggregated.
• FedRep: Learns a global representation (all layers except the head) collaboratively. Each client trains a unique local head (classifier) in isolation. FedRep alternates between local head updates and global representation updates.
• Advantage: Provides explicit architectural control over the personalization granularity and reduces communication costs.

Clustered federated learning is a PFL-adjacent approach where clients are partitioned into clusters based on the similarity of their data distributions. A separate global model is learned for each cluster, which acts as a personalized model for all clients within that cluster.

• Process: The server uses client update similarities (e.g., cosine similarity of gradient vectors) to perform online clustering. Aggregation (e.g., FedAvg) is then performed within each cluster.
• Use Case: Effective when client populations naturally form groups (e.g., different geographic regions, device types). It provides coarse-grained personalization between clusters while maintaining collaboration within them.

Meta-learning for PFL applies algorithms like Model-Agnostic Meta-Learning (MAML) to learn a global model initialization that can be rapidly adapted to new clients with only a few steps of local training. Personalized Federated Meta-Learning (PFML) is a direct implementation of this idea.

• Objective: Learn a global model θ such that for any client i, performing a few gradient steps on local data yields a high-performance personalized model θᵢ.
• Mathematical Goal: θ* = argmin θ Σᵢ Lᵢ( Alg(θ, Dᵢ) ), where Alg is a local adaptation procedure like SGD.
• Benefit: Explicitly optimizes the global model for fast personalization, making it highly sample-efficient for new clients.

A Mixture of Experts (MoE) for PFL is an architecture where a global set of expert sub-models {E₁, E₂, ..., Eₖ} is learned collaboratively. Each client has a local, lightweight gating network that learns to select or combine these experts to form a personalized model for local inference.

• Inference: For an input x, client i's output is a weighted sum: Σⱼ gᵢ(x)ⱼ * Eⱼ(x), where gᵢ is client i's gating network.
• Training: Experts are trained on all client data (with routing). Gating networks are trained locally and privately.
• Advantage: Enables high-capacity, specialized modeling while maintaining parameter efficiency and personalization flexibility.