FedPer

The core innovation of FedPer is its split neural network architecture. The model is divided into two distinct components:

• Base Layers (Body): Lower, foundational layers (e.g., convolutional feature extractors, transformer encoders) that are trained collaboratively across all clients and aggregated by the server.
• Personalization Layers (Head): The final layers (typically one or more fully connected layers) that are unique to each client, trained only on local data, and never shared. This separation allows the model to learn generalizable features globally while adapting the final decision-making to individual client data distributions.

FedPer directly combats client drift, a major challenge in federated learning with non-IID data. By keeping the personalized head local, clients are not forced to converge their final representations to a single global average that may be suboptimal for their unique data. The local head can specialize, reducing the conflict between local optimization and the global objective. This leads to:

• Faster convergence for personalized tasks.
• Higher final accuracy on local client data.
• More stable global base layer training, as the aggregation is focused on shared feature representations rather than task-specific outputs.

FedPer operates through a structured, two-phase training process within each communication round:

1. Collaborative Base Learning: The server broadcasts the global base layers to selected clients. Clients freeze their local personalization layers and perform several epochs of training only on the base layers using their local data. The updates to these base layers are sent back to the server for federated averaging.
2. Local Personalization: Concurrently or sequentially, each client unfreezes and trains its personalization layers on its local data. These updates are never communicated, ensuring the personalized knowledge remains private on the device. This protocol cleanly separates the learning of shared features from personalized adaptation.

FedPer enhances privacy beyond standard federated learning through architectural constraints. Since the personalization layers and their gradients never leave the client device, the most sensitive part of the model—the part directly mapping learned features to user-specific labels or actions—is never exposed. This provides a strong privacy guarantee:

• The server only ever receives updates for the generalized base layers.
• It is computationally infeasible for the server or other clients to reconstruct a user's unique data patterns from the aggregated base model updates alone. This makes FedPer particularly suitable for applications like next-word prediction, content recommendation, and healthcare diagnostics where user behavior is highly individual.

FedPer established a blueprint for Personalized Federated Learning (PFL). Key related approaches include:

• FedRep: Extends FedPer by explicitly learning a shared representation (base layers) and cycling between representation and head training.
• Per-FedAvg: A meta-learning approach that aims to find a global model initialization that can be quickly personalized with few local steps, rather than maintaining separate layers.
• Local Fine-Tuning: A simpler baseline where a global model is trained via FedAvg and then each client fine-tunes the entire model locally. FedPer is more communication-efficient than this, as only the base layers require aggregation. FedPer's clear separation of concerns makes it a versatile and interpretable starting point for PFL research.

Implementing FedPer requires careful system design:

• Model Architecture Choice: The split point between base and personalization layers is a hyperparameter. Deeper splits offer more personalization but may reduce the benefit of collaborative learning.
• Client Storage: Each client must store its unique personalization head in addition to the global base model.
• Cold Start for New Clients: A new client joining the system only receives the global base layers. It must train its personalization head from scratch on its local data, which may require sufficient local data for effective personalization.
• Compatibility with Secure Aggregation: The aggregation of base layer updates is fully compatible with cryptographic secure aggregation protocols, providing an additional layer of privacy.

FedPer is ideal for training diagnostic models across multiple hospitals without sharing sensitive patient data. The shared base layers learn general visual features from diverse medical imaging data (e.g., X-rays, MRIs) collaboratively. Each hospital's personalized head then fine-tunes predictions to its specific patient demographics, imaging equipment, and local disease prevalence. This addresses statistical heterogeneity (non-IID data) between institutions while preserving privacy under regulations like HIPAA and GDPR.

Smartphone keyboard applications use FedPer to improve typing suggestions and autocorrect. The global base model learns universal language patterns and common grammar from aggregated, anonymized updates. The local personalized layer adapts to the individual user's unique vocabulary, slang, typing style, and frequently used phrases (e.g., contact names, work jargon). This provides a responsive, personalized user experience without transmitting private keystroke data to a central server.

In manufacturing, FedPer enables predictive maintenance models for fleets of machines. The collaborative base learns general failure signatures from vibration, temperature, and acoustic data across all machines. A machine-specific head then personalizes predictions for each unit's unique operating conditions, wear patterns, and maintenance history. This results in more accurate failure forecasts for individual assets while keeping proprietary operational data on-premise at each factory.

Banks can collaborate to detect novel fraud patterns without exposing customer transaction data. FedPer's shared representation layers identify emerging, complex fraud tactics from aggregated signals. Each bank's personalized classifier then tailors detection to its specific customer base, product mix, and regional risk factors. This improves detection of both global fraud campaigns and institution-specific anomalies, enhancing security while maintaining strict financial data sovereignty.

Streaming services and news platforms use FedPer to refine recommendation engines. The global model body learns broad content embeddings and general user preference patterns. The local user head, trained exclusively on the device, captures deep personal tastes, temporal context (time of day), and changing interests. This architecture mitigates the cold-start problem for new users via the shared base while providing highly adaptive recommendations without centralizing sensitive viewing history.

Research consortia in fields like genomics or material science employ FedPer. Participants jointly train a foundational model on heterogeneous, proprietary datasets to discover general scientific principles. Each research lab then attaches a specialized output layer to solve its specific experimental or diagnostic task. This accelerates foundational discovery while protecting intellectual property and allowing for domain-specific application, a key advantage over methods like FedAvg that force a single global output.

The foundational algorithm for federated learning. A central server coordinates multiple clients to train a shared global model by periodically averaging their locally computed model updates. It establishes the core federated training loop of broadcast, local training, and aggregation.

• Core Mechanism: Clients perform multiple local epochs of SGD. The server aggregates updates via weighted averaging based on local dataset sizes.
• Challenge: Prone to client drift under statistical heterogeneity (non-IID data), as local models diverge from the global objective.

An optimization algorithm designed to mitigate client drift caused by data heterogeneity. It modifies the local client objective function by adding a proximal term.

• Mechanism: The proximal term penalizes the local model's deviation from the global model received at the start of the round. This constraint keeps client updates closer to the global optimum.
• Use Case: Particularly effective in cross-device settings with highly variable data distributions and system capabilities.

A personalized federated learning approach, similar in spirit to FedPer, where the model is split into a shared representation (body) and a client-specific head.

• Key Difference vs. FedPer: In FedRep, clients alternate between learning the shared representation and their personal head within each communication round. FedPer typically learns them concurrently or in distinct phases.
• Outcome: Enables collaborative learning of general features while preserving user-specific decision layers.

A method that handles feature distribution shift across clients by localizing batch normalization layers. Instead of aggregating them, each client maintains its own batch norm statistics (mean and variance).

• Mechanism: Only convolutional/linear layer weights are averaged by the server. Local BN stats capture client-specific data characteristics.
• Benefit: Significantly improves performance on non-IID data without adding complex personalization layers, acting as a lightweight form of personalization.

An algorithm that uses control variates (correction terms) to reduce the variance between client updates, achieving faster and more stable convergence under non-IID data.

• Core Idea: Maintains a global control variate on the server and a local control variate on each client. The difference between them corrects for client drift.
• Advantage: Provides theoretical convergence guarantees under heterogeneous data, often converging in fewer communication rounds than FedAvg.

The overarching goal of producing a model tailored to an individual client's data distribution. FedPer is a specific technique within this category.

• Common Techniques:
  • Local Fine-Tuning: Taking the global model and fine-tuning it locally post-training.
  • Multi-Task Learning: Framing each client's problem as a related task.
  • Mixture of Experts: Using a gating network to combine specialized models.
  • Meta-Learning (e.g., Per-FedAvg): Learning a model initialization that can adapt quickly with few local steps.
• Trade-off: Balances global knowledge aggregation with local specialization.