Personalized Federated Learning

The most common PFL paradigm, where a shared global model is first trained collaboratively and then adapted to each client's local data. This is often achieved through fine-tuning on the client side after federated training concludes.

• Mechanism: A global model is trained using standard Federated Averaging (FedAvg). Each client then performs a few steps of local training on its private dataset to shift the model toward its specific distribution.
• Key Benefit: Simple to implement and leverages the generalizable features learned from the broader population.
• Trade-off: The global model may be a poor initialization for clients with highly divergent data, a problem known as weight divergence.

This approach frames federated personalization as a multi-task learning problem, treating each client's local optimization as a distinct but related task. The goal is to learn personalized models that are regularized to be close to each other in parameter space.

• MOCHA Algorithm: A representative framework that uses a primal-dual optimization method to handle statistical and systems heterogeneity simultaneously, learning a separate model for each client while capturing relationships between tasks.
• Regularization: Techniques like Moreau envelopes (used in pFedMe) decouple personalized model optimization from the global model, preventing the local model from straying too far from a beneficial global prior.
• Benefit: Explicitly models the relationships between clients, offering strong theoretical guarantees for convergence.

A personalized model is constructed as a mixture of a global model and a purely local model, balancing general knowledge with client-specific adaptation. The mixing weight is a critical hyperparameter.

• APFL (Adaptive Personalized FL): Learns an optimal mixing coefficient, α, for each client to combine the predictions of a global model and a local model, adapting to the degree of local statistical shift.
• L2GD (Local and Global Gradient Descent): Explicitly separates the optimization step into a local descent direction and a global descent direction, allowing the client to control the trade-off between personalization and generalization.
• Key Insight: Prevents the local model from overfitting to a small, biased local dataset by anchoring it to the globally learned representation.

This strategy partitions the neural network architecture into shared base layers and personalized head layers. The base layers, which learn generic feature extractors, are federated, while the head layers, which learn client-specific classification or regression logic, remain purely local.

• FedPer: A foundational approach where base layers are aggregated globally, and the final classification layers are personalized and never shared. This aligns with the intuition that high-level features are general, while decision boundaries are personal.
• FedRep: Extends this by learning a shared low-dimensional data representation across clients and a unique local classifier head for each client.
• Communication Efficiency: Only the shared base layers need to be transmitted, reducing the communication payload compared to full-model aggregation.

Instead of a single global model, this method partitions the client population into clusters of clients with similar data distributions. A separate model is trained for each cluster, providing group-level personalization.

• Hypothesis: A single global model is suboptimal for a population with distinct modes in its data distribution. Clustering identifies these modes.
• CFL (Clustered FL): Iteratively separates clients based on the cosine similarity of their gradient updates. Clients whose models are being pulled in different directions are assigned to different groups.
• Application: Ideal for scenarios like a network of IoT sensors where devices in different geographic locations experience fundamentally different environmental conditions.

Also known as Federated Model-Agnostic Meta-Learning (FedMeta), this approach does not aim to learn a single predictive model but rather a good initialization that can be rapidly adapted to a new client's data with only a few steps of gradient descent.

• Per-FedAvg: A variant of FedAvg that modifies the local update step to explicitly optimize for a model that is easy to personalize, using a gradient of the local gradient (a Hessian-vector product).
• Reptile Algorithm: A simpler first-order meta-learning algorithm applied in a federated context, where the global model is updated by interpolating toward each client's adapted model.
• Goal: Minimize the number of local training steps required for a new client to achieve high performance, critical for cross-device FL with ephemeral participation.

The fundamental challenge that Personalized Federated Learning is designed to solve. In real-world deployments, local client data is almost never independently and identically distributed (Non-IID). A user's typing patterns, a factory's acoustic profile, or a hospital's patient demographics create distinct local distributions. Training a single global model on such data forces a compromise that degrades performance for individual clients. Statistical heterogeneity causes local optimization objectives to drift away from the global optimum, a phenomenon known as client drift. Personalization techniques explicitly model this diversity rather than treating it as noise, allowing each client's model to specialize to its own data manifold while still benefiting from shared knowledge.

The foundational aggregation algorithm that serves as the baseline for most personalized variants. FedAvg operates by:

• Selecting a subset of clients for each training round
• Having each client perform multiple local stochastic gradient descent (SGD) steps on its own data
• Transmitting only the updated model weights back to the server
• Computing a weighted average of these updates to form the new global model

While effective for homogeneous data, FedAvg's single-model approach struggles under Non-IID conditions. Personalized methods like FedProx and SCAFFOLD modify this core loop by adding proximal terms or control variates to correct for client drift during local training.

A communication-efficient alternative to weight-based aggregation that aligns naturally with personalization goals. Instead of averaging model parameters, clients exchange model outputs (logits) on a shared public dataset or a set of unlabeled samples. The server uses knowledge distillation to transfer the collective knowledge into a global model, while clients can use the same technique to co-distill personalized models.

Key advantages for personalization:

• Supports heterogeneous model architectures across clients, allowing each device to use a model sized for its compute budget
• Exchanging logits instead of weights provides an implicit form of privacy
• Naturally handles divergent local objectives since only soft labels are shared, not parameter directions

A meta-learning framework that has been adapted for personalized federated settings under the name Per-FedAvg. The core idea is to find an initial global model that can be rapidly adapted to any client's local data distribution with just a few gradient steps. The training process involves:

• Computing the meta-gradient by differentiating through the local adaptation process itself
• Optimizing the global model so that it serves as an excellent initialization point for personalization
• Each client performs a few steps of fine-tuning on its local data to create its personalized model

This approach explicitly optimizes for fast adaptability rather than global accuracy, making it ideal for environments with extreme statistical diversity.

A framework that treats each client's learning problem as a distinct but related task within a multi-task learning system. Rather than forcing all clients toward a single consensus model, this approach learns a shared representation while allowing task-specific parameters to diverge. Common architectures include:

• MOCHA: A distributed multi-task learning algorithm that uses a primal-dual optimization approach to handle communication constraints and stragglers
• Clustered Federated Learning: Groups clients with similar data distributions into clusters, training a separate model for each cluster
• Federated Learning with Personalization Layers: Keeps base layers shared across all clients while allowing personalization layers near the output to specialize locally

This approach explicitly models the relationships between client distributions rather than ignoring them.

A mathematical framework that provides provable privacy guarantees for the personalized model updates transmitted from clients. Differential privacy works by injecting calibrated noise into the model updates before transmission, ensuring that the presence or absence of any single training example is statistically indistinguishable. The privacy budget is controlled by the parameter epsilon (ε)—lower values provide stronger privacy but may degrade personalization quality.

In personalized federated learning, differential privacy can be applied at multiple levels:

• Local differential privacy: Each client perturbs its own update before sending
• Central differential privacy: The server adds noise during aggregation
• Personalized differential privacy: Allowing clients to choose their own privacy budgets based on their sensitivity requirements