Layer-wise Personalization

The core mechanism involves a hard architectural split within the neural network model. Typically, the base layers (e.g., convolutional feature extractors, transformer encoders) are designated as global layers. These are trained via federated averaging and shared across all clients. In contrast, the head layers (e.g., final fully-connected classifiers, task-specific output layers) are designated as personalization layers. These remain local, are never aggregated, and are trained solely on the client's private data. This partition is defined before training begins.

This strategy inherently decouples learning objectives. The global layers learn a universal feature representation that is broadly useful across all client data distributions. The personalization layers then learn a client-specific decision function that maps these shared features to local labels. For example, a global model might learn to identify edges and shapes in images, while a personal head learns to classify those features into 'cat' or 'dog' based on a user's unique pet photos.

It directly addresses covariate shift and feature distribution shift, common in non-IID data. By keeping normalization layers (e.g., BatchNorm, LayerNorm) local—a technique exemplified by FedBN—the model adapts to the local statistical properties of a client's feature activations. This prevents the degradation that occurs when aggregating mismatched feature statistics across heterogeneous clients, leading to more stable and accurate personalized models.

It is highly communication-efficient. During each federated round, clients only transmit updates for the global layers to the server, significantly reducing the payload size compared to sending entire model weights. From a privacy perspective, it provides a structural guarantee that the most task-specific parameters, which are most reflective of the local data distribution, never leave the device. This reduces the risk of inference attacks on the aggregated model.

FedPer is the foundational algorithm for this paradigm, keeping base layers global and the classification head personal. FedRep extends this by explicitly framing it as learning a global representation and local heads. The training loop follows a specific pattern:

• Server Step: Aggregates updates for global layers from clients.
• Client Step: Freezes personal layers, trains global layers on local data.
• Personalization Phase: Freezes updated global layers, trains personal layers on local data.

The primary trade-off is between flexibility and architectural control. Unlike methods like local fine-tuning, which allow any parameter to change, layer-wise personalization imposes a rigid structure. This can be advantageous for system design and debugging but may be suboptimal if the ideal personalization boundary does not align with the pre-defined layer split. It is less flexible than per-parameter or meta-learning approaches but offers greater interpretability and training stability.

FedPer is the foundational algorithm for layer-wise personalization. It establishes the core architectural pattern:

• Global Base Layers: The initial layers (feature extractor) are trained collaboratively via federated averaging.
• Local Head Layers: The final classification or regression layers are kept entirely local, trained only on the client's private data, and never shared with the server. This clean separation allows the model to learn general feature representations globally while enabling each client to learn a decision boundary perfectly suited to its local data distribution. It is highly effective for tasks where the feature space is similar across clients, but the label distributions differ.

FedRep (Federated Representation Learning) refines the FedPer concept by explicitly decoupling the learning phases:

• Phase 1 - Representation Learning: Clients collaboratively train only the global representation layers (the backbone) while their local heads are frozen.
• Phase 2 - Head Personalization: The global backbone is frozen, and each client trains only its unique local head on its private data. This alternating training ensures the global representation is learned without being biased by the heterogeneous local tasks. It is particularly well-suited for scenarios with significant label distribution skew (non-IID data).

FedBN (Federated Batch Normalization) addresses feature distribution shift—a common challenge where data statistics (e.g., image brightness, sensor calibration) vary across clients. Its key mechanism:

• Global Weights: Convolutional and linear layer weights are aggregated federated.
• Local Statistics: The mean and variance parameters within Batch Normalization (BN) layers are not averaged. Each client maintains and uses its own BN statistics during local training and inference. This simple yet powerful technique allows the model to normalize features according to each client's local data distribution, significantly improving performance on heterogeneous data without adding complex communication overhead.

This is a generalized, configurable pattern where an arbitrary number of layers are designated as personalizable.

• Architectural Control: The system designer selects which contiguous block of layers (e.g., the first K layers, the last M layers, or a specific middle module) will be local.
• Use Case Driven: The choice is task-dependent. Making early layers local can help with domain-specific low-level feature adaptation (e.g., sensor noise). Making later layers local is better for task-specific decision adaptation.
• Hyperparameter (K): The depth of personalization becomes a critical hyperparameter, trading off between individual adaptation and collaborative learning benefits. This pattern is common in production PFL systems for its flexibility.

Instead of personalizing native model layers, this pattern inserts small, trainable adapter modules into a frozen global model.

• Frozen Global Backbone: The primary model weights are trained federated and then frozen.
• Local Adapters: Lightweight neural modules (e.g., bottleneck layers) are added to specific points in the network. Only these adapters are trained locally and kept private.
• Extreme Efficiency: This approach minimizes local compute and storage footprint, as adapters contain a tiny fraction of the model's total parameters. It is inspired by Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA and is ideal for deployment on resource-constrained edge devices.

This advanced pattern combines architectural separation with optimization-based personalization.

• Meta-Initialized Global Model: A model is trained via Model-Agnostic Meta-Learning (MAML) or Reptile in a federated setting, resulting in a global initialization that is highly adaptable.
• Rapid Local Adaptation: When a client receives this model, it performs fast adaptation (a few gradient steps) specifically on its designated personalization layers.
• Synergistic Effect: The meta-learning provides a head start in the parameter space, and the layer-wise constraint ensures adaptation is efficient and focused. This is effective for cross-silo PFL where clients have sufficient data for meaningful few-shot adaptation.

Personalized Federated Learning (PFL) is a decentralized machine learning paradigm where a global model is collaboratively trained across multiple clients, but the final deployed model is tailored or adapted to the unique data distribution of each individual client. The core challenge is balancing the benefits of collaborative learning from pooled data with the need for models that perform well on local, non-IID data.

• Goal: Produce a set of models {θ₁, θ₂, ..., θₙ} for N clients, each optimized for its local data distribution Pᵢ.
• Contrast with Standard FL: Standard FL aims for a single global model θᴳ; PFL explicitly seeks personalized variants.
• Primary Use Case: Applications with significant statistical heterogeneity across clients, such as next-word prediction on different smartphones or health monitoring across diverse patient populations.

A personalized model head is the task-specific final layer(s) of a neural network that are unique to each client in a PFL system, while the earlier feature-extraction layers are shared and trained federated. This is the most common instantiation of layer-wise personalization.

• Architecture: The model is split into a global body (convolutional layers, transformers blocks) and a local head (fully-connected classification/regression layers).
• Training: The global body is aggregated via FedAvg. The local head is trained only on the client's private data and never shared.
• Examples: In FedPer and FedRep, the head is the personalized component. For image classification, the head is typically the final classifier; for language models, it could be the output projection layer.

FedPer and FedRep are two canonical PFL algorithms that implement layer-wise personalization by decoupling shared representation learning from client-specific adaptation.

• FedPer (Federated Personalization): Keeps base layers global and the final classification layers personal. The personalized layers are updated locally and never aggregated on the server.
• FedRep (Federated Representation): Learns a global representation (all layers except the last) across clients. Each client learns a unique local head (the final layer). FedRep alternates between local head updates and global representation updates.
• Key Difference: FedPer aggregates the global base layers. FedRep's training loop explicitly separates representation and head updates, often leading to more robust shared features.

Local fine-tuning is a post-aggregation PFL technique where a global model θᴳ, received from the federated server, is further trained on a client's local dataset Dᵢ to adapt it to the client's specific data distribution before deployment.

• Process: θᵢ = LocalTrain(θᴳ, Dᵢ, η, τ), where η is the learning rate and τ is the number of local epochs.
• Advantage: Simple to implement; requires no changes to the core federated aggregation protocol.
• Risk: Can lead to client drift, where excessive local training causes θᵢ to diverge too far from the useful knowledge in θᴳ, potentially harming performance. Often used with regularization (e.g., penalizing ||θᵢ - θᴳ||²) to mitigate this.

Client-specific models are the end-product of any PFL process, referring to the unique model instances {θ₁, θ₂, ..., θₙ} that are optimized for the statistical properties of an individual client's data. They are derived from a shared global foundation.

• Not a Monolith: Can be generated via different mechanisms: unique architectures (personalized heads), unique parameters (via fine-tuning), or unique combinations of components (MoE).
• Evaluation: Performance is measured by the average accuracy of each θᵢ on its own local test set, not on a global holdout set.
• Deployment Challenge: Requires managing and serving N potentially different model artifacts, complicating ML operations compared to a single global model.

Multi-Task Federated Learning (MTFL) frames the PFL problem as a multi-task learning problem, where each client's learning objective is treated as a related but distinct task. The goal is to learn a set of models that share knowledge while performing well on their individual tasks.

• Theoretical Foundation: Provides a formal framework for understanding data heterogeneity as task heterogeneity.
• Methods: Includes learning shared and task-specific parameters, using regularization to encourage parameter similarity, or modeling relationships between client tasks.
• Relation to Layer-wise Personalization: MTFL often results in architectures with shared layers (common knowledge) and task-specific/personalized layers, making it a theoretical superset of the layer-wise approach.