Personalized Federated Averaging (pFedAvg)

The defining characteristic of pFedAvg is the post-aggregation personalization step. After receiving the global model from the server, each client performs additional local training epochs exclusively on its own data. This local fine-tuning adapts the general global model to the client's specific data distribution, creating a unique personalized model for inference.

• Key Distinction: Unlike FedAvg where all clients deploy the same global model, pFedAvg results in a model per client.
• Example: A global next-word prediction model, after pFedAvg, becomes specialized for a user's unique writing style and vocabulary on their personal device.

pFedAvg explicitly separates two optimization goals: learning a shared global representation and achieving local task performance. The federated averaging phase learns common features, while the local personalization phase optimizes for the client's private objective.

• This mitigates client drift, where local SGD steps on heterogeneous data can pull models in conflicting directions, harming global convergence.
• The algorithm balances model generalization (from federation) with model specialization (from local adaptation).

pFedAvg is fundamentally designed for Non-Independent and Identically Distributed (Non-IID) client data, the rule rather than the exception in federated settings. Client data distributions vary significantly (e.g., different user photo libraries, local sensor readings, personal text messages).

• Standard FedAvg often performs poorly on Non-IID data as a single global model cannot fit all distributions.
• pFedAvg directly addresses this by allowing the final model to diverge to fit local data patterns after benefiting from shared learning.

The pFedAvg training process can be framed as a bi-level optimization problem. The inner loop is the local personalization (fine-tuning) on each client. The outer loop is the federated aggregation of global model parameters across clients.

• Outer Loop (Server): Aggregates client updates to improve the global starting point for all.
• Inner Loop (Client): Takes the global model and optimizes it for local loss.
• This structure is a precursor to more advanced meta-learning approaches like Per-FedAvg (Personalized Federated Meta-Learning).

pFedAvg introduces a trade-off between communication efficiency and personalization quality. It typically uses the same communication pattern as FedAvg (sending full model updates) but changes the computation pattern.

• Increased Local Compute: Clients perform more local epochs per round for personalization, increasing on-device computation.
• Unchanged Communication: The communication cost per round is similar to FedAvg, as the same global model size is transmitted.
• The benefit is reduced personalization latency at deployment; the model is already adapted, requiring no further fine-tuning when used.

pFedAvg is a foundational, flexible framework that has inspired numerous specialized Personalized Federated Learning (PFL) algorithms. Many advanced methods can be viewed as enhancements to its core two-phase idea.

• FedPer/FedRep: Architecturally enforce personalization by keeping specific layers (e.g., the classification head) local and never aggregating them.
• pFedMe: Uses a Moreau envelope as a regularized local loss to explicitly separate personalization from global learning.
• Per-FedAvg: Applies Model-Agnostic Meta-Learning (MAML) principles to learn a global model explicitly designed for fast personalization.

FedPer is a foundational pFedAvg variant that architecturally separates a neural network into base layers and personalized layers. The base layers (feature extractor) are trained collaboratively via federated averaging. The personalized layers (typically the final classification head) remain local to each client, are trained only on local data, and are never aggregated. This creates personalized models where feature representations are shared, but decision boundaries are unique.

• Key Mechanism: Architectural separation of global and local parameters.
• Use Case: Ideal for tasks where clients share low-level features (e.g., image textures) but have distinct label distributions.

This is the simplest pFedAvg technique. After the federated training process converges to a global model, each client downloads this model and performs additional local fine-tuning using its private dataset. This adapts the general model to the client's specific data distribution.

• Process: FedAvg → Global Model → Client-Side Fine-Tuning → Personalized Model.
• Advantage: Extremely simple to implement atop any FL framework.
• Risk: Can lead to catastrophic forgetting of useful shared knowledge if fine-tuning is too aggressive.

pFedMe reformulates the PFL objective using a Moreau envelope, which explicitly separates the global model learning from local personalization. Each client solves a bi-level optimization problem: it finds a personalized model close to its local optimum while regularizing it to be near a global reference model. The server then aggregates these global reference models.

• Core Innovation: Uses a regularized loss function: F_i(w_i) + (λ/2) ||w_i - θ||^2, where w_i is the local model and θ is the global reference.
• Benefit: Provides stronger theoretical convergence guarantees under data heterogeneity compared to simple fine-tuning.

Per-FedAvg applies Model-Agnostic Meta-Learning (MAML) principles within federated learning. Instead of learning a single global model, the algorithm learns a global model initialization that is explicitly optimized for fast adaptation. Each client can then rapidly personalize this initialization with only one or a few steps of gradient descent on its local data.

• Objective: min_θ Σ_i F_i(θ - α ∇F_i(θ)). The goal is to find an initialization θ such that a single local step yields a good personalized model.
• Result: The global model acts as a strong prior, enabling efficient personalization even for clients with very limited data.

Federated Batch Normalization (FedBN) is a simple yet effective pFedAvg variant for convolutional neural networks. It identifies that Batch Normalization (BN) layer statistics (mean and variance) are highly sensitive to local data distribution. In FedBN, BN parameters are not aggregated with the server; they remain local and are updated only using the client's own data during training.

• Mechanism: All non-BN parameters are federated. All BN parameters are kept client-specific.
• Impact: Effectively counters feature shift across clients with minimal architectural change and communication overhead.

While not pFedAvg in the strictest sense, clustered FL is a direct precursor and enabler. It addresses heterogeneity by partitioning clients into clusters based on data distribution similarity (e.g., via gradient directions or model performance). A separate global model is then learned for each cluster via FedAvg.

• Process: 1. Cluster clients. 2. Run parallel FedAvg within each cluster. 3. Clients personalize within their cluster's model.
• Outcome: Clients receive a model that is already specialized to their data 'type', requiring less aggressive personalization.

The foundational algorithm for standard federated learning. FedAvg coordinates learning across decentralized devices by:

• Having clients perform local stochastic gradient descent on their private data.
• Transmitting only the resulting model updates (parameter deltas) to a central server.
• The server aggregates these updates via a weighted average to form a new global model.
• This global model is then redistributed for the next round.

pFedAvg modifies this core loop to produce client-specific models instead of a single global one.

A straightforward post-hoc personalization technique. After the federated training process concludes and a global model is distributed, each client performs additional training steps exclusively on its local dataset. This adapts the general model to the client's specific data distribution. It's a common baseline method and a key component within more sophisticated PFL algorithms like pFedAvg, where fine-tuning can occur interleaved with global rounds.

An architectural strategy for PFL where specific components of a neural network are designated as client-specific. Common implementations include:

• FedPer: Keeps the base layers (feature extractor) global and federated, while the final classification head is local and personal.
• FedRep: Learns a global representation (feature extractor) collaboratively, but trains unique local heads for each client.

This approach decouples shared feature learning from client-specific decision boundaries, providing a structured way to implement personalization.

A critical challenge in PFL and non-IID federated learning. Client drift occurs when local training on statistically heterogeneous data causes client models to diverge significantly from the global objective. This divergence can:

• Slow down or prevent global convergence.
• Reduce the utility of the global model for personalization.

pFedAvg and related algorithms implicitly or explicitly incorporate mechanisms, such as regularization or controlled aggregation, to compensate for and manage this drift for beneficial personalization.

A PFL strategy where a client's final model is a weighted combination of multiple models. A common formula is: θ_personalized = α * θ_local + (1-α) * θ_global

Where:

• θ_local is a model trained solely on the client's private data.
• θ_global is the collaboratively trained federated model.
• α is a mixing hyperparameter controlling the personalization degree.

This method allows smooth control between a fully local model and the global model, balancing specificity with shared knowledge.

Applies meta-learning principles to learn a global model that is explicitly easy to personalize. Algorithms like Model-Agnostic Meta-Learning (MAML) are adapted to the federated setting. The goal is to learn a global model initialization that can be rapidly adapted to a new client's data with only a few gradient steps. Personalized Federated Meta-Learning (PFML) frameworks formalize this, aiming to produce a meta-model that maximizes average performance after fast local adaptation across all clients.