FedPer

FedPer's core innovation is the architectural separation of a neural network into two distinct sets of layers:

• Base Layers (Global): The initial layers (e.g., convolutional feature extractors, transformer encoders) are trained collaboratively across all clients. These layers learn a general-purpose data representation that is useful for all participants.
• Personalized Layers (Local): The final layers (typically the classification or regression head) remain exclusively on each client's device. These layers are trained only on local data and are never aggregated or shared with the server.

This partition creates a clear boundary between shared knowledge (general features) and private adaptation (task-specific decision boundaries).

Training in FedPer occurs in two distinct, alternating phases within each communication round, enforcing the separation of concerns:

1. Local Representation Learning: Clients receive the latest global base layers from the server. They perform local stochastic gradient descent (SGD) on their private data, updating both the base layers and their local personalized layers. The gradients for the base layers are computed to improve the feature extractor for the client's specific data distribution.
2. Global Aggregation: After local training, clients send only their updated base layer parameters to the server. The server performs Federated Averaging (FedAvg) on these base layers to create a new global feature extractor. The personalized layers are not transmitted, preserving their unique, client-specific nature.

This decoupling allows the global model to improve from diverse data while personalization happens autonomously on-device.

FedPer provides inherent data privacy advantages beyond standard federated learning:

• No Exposure of Personalized Parameters: Since the personalized head is never shared, the server gains zero knowledge about the client's final decision function, which is most directly reflective of their private label distribution.
• Limited Signal from Base Layers: While base layers are shared, they represent a more abstract, general feature space. Inferring raw data or specific labels from these aggregated parameters is significantly more difficult than from a full model.
• Reduced Attack Surface: Techniques like model inversion or membership inference attacks targeting the personalized behavior are infeasible against the server, as the critical personalized component is never present there.

This makes FedPer suitable for scenarios with high sensitivity in the output space or label distribution.

FedPer is explicitly designed for the non-independent and identically distributed (non-IID) data reality of federated learning. It mitigates client drift—where local updates pull the global model in conflicting directions—through its architecture:

• Global Base Layers: Learn robust, general features that are invariant across different client data distributions (e.g., edges, shapes, basic semantics in images).
• Local Personalized Layers: Capture the client-specific distribution shift. For example, one client's "car" images might be mostly sedans, while another's are trucks. The personalized head learns the unique decision boundary for that client's subset of classes.

This approach often outperforms a single global model on non-IID data, as the global model no longer needs to compromise on a one-size-fits-all classifier.

FedPer can be more communication-efficient than training full models locally and can reduce local compute for personalization:

• Reduced Upload Payload: Clients transmit only the parameters of the (typically smaller) base layers, not the entire model. For large models where the head is small relative to the base (e.g., large vision models), the savings are modest, but for models with large heads, savings are significant.
• Faster Local Adaptation: During inference or further local fine-tuning, only the (smaller) personalized head may need adjustment, requiring less computation than fine-tuning a full model.
• Stable Global Convergence: By focusing aggregation on the feature extractor, FedPer can lead to more stable global training compared to aggregating full models on highly heterogeneous data, potentially requiring fewer communication rounds to reach a useful shared representation.

FedPer's clear architectural template has inspired and serves as a foundation for numerous advanced personalized federated learning algorithms:

• FedRep: Extends FedPer by alternating phases of learning only the local head while freezing the base, and then learning only the base while freezing the head, further decoupling the tasks.
• Layer-Wise Personalization Strategies: FedPer's principle informs decisions on which layers to personalize. For example, later transformer blocks in NLP might be personalized while early ones are kept global.
• Hybrid Approaches: FedPer can be combined with meta-learning (learning a better global base initialization) or regularization (e.g., penalizing the distance between local and global base layers) for improved performance.

It provides a versatile and interpretable starting point for PFL system design.

FedPer is ideal for training diagnostic models across multiple hospitals without sharing sensitive patient data. A global feature extractor learns common patterns from medical images (e.g., X-rays, MRIs) across institutions, while each hospital trains a local personalized head optimized for its specific patient demographics, imaging equipment, and prevalent conditions. This ensures high accuracy for local cases while preserving patient privacy and complying with regulations like HIPAA and GDPR.

• Example: A model for detecting diabetic retinopathy where the global layers learn general retinal features, and local heads at clinics in different regions specialize in variations linked to specific ethnic populations or local camera hardware.

Smartphone keyboards use FedPer to improve autocorrect and next-word prediction without uploading personal typing data. A global model learns universal language syntax and common phrases. On-device, the personalized head adapts to the user's unique vocabulary, slang, abbreviations, and frequently used contacts' names. This provides a highly responsive, private user experience.

• Key Benefit: Drastically reduces communication overhead compared to sending full model updates, as only the global base layers are aggregated. The personalized parameters never leave the device.

Banks and financial institutions can collaboratively improve fraud detection models. FedPer allows a global model to learn universal patterns of fraudulent transactions. Each bank's local head then personalizes the model to its specific customer base, regional transaction types, and localized fraud tactics. This addresses the non-IID data challenge—fraud patterns in one region may differ significantly from another.

• Privacy Guarantee: Sensitive transaction histories never leave the bank's secure servers. Only updates to the anonymized, global feature layers are shared, protecting both customer data and proprietary risk models.

In manufacturing, FedPer enables predictive maintenance models for fleets of machines from different factories or with varying operating conditions. A global model learns shared failure signatures from vibration, temperature, and acoustic sensor data. Each factory's local head personalizes the model for its specific machine models, wear patterns, and local environmental factors (e.g., humidity, dust).

• Result: More accurate failure predictions for each asset, leading to reduced downtime. The system learns from the collective experience of all factories without exposing proprietary operational data.

Streaming services and news platforms use FedPer to build recommendation systems that respect user privacy. A global model learns general user-item interaction patterns and content embeddings. On each user's device, a local recommendation head is trained on that individual's watch history, clicks, and dwell times, creating a hyper-personalized feed.

• Advantage Over Centralized Training: Mitigates the cold-start problem for new users by leveraging the global feature knowledge, while quickly adapting locally. It also prevents the server from building a complete, intrusive profile of any single user's preferences.

FedPer can enhance behavioral biometrics for continuous authentication. A global model learns common behavioral patterns (typing rhythm, swipe gestures, gait from phone sensors). A highly personalized local head on each user's device fine-tunes to that individual's unique behavioral nuances, creating a robust and private authentication model.

• Security Property: The model authenticates based on implicit behavioral signals. The personalized head, which contains the most sensitive biometric template, never leaves the user's device, making the system resilient to server-side breaches and model inversion attacks.

The overarching paradigm where a global model is trained collaboratively across clients, but the final deployed model is tailored to each client's unique data distribution. Key goals include:

• Overcoming statistical heterogeneity (non-IID data).
• Improving local model accuracy without compromising data privacy.
• Enabling applications like next-word prediction and health monitoring where user patterns vary significantly. PFL contrasts with standard federated learning, which aims for a single global model that performs well on average.

The task-specific final layer(s) of a neural network kept local to each client. This is the core mechanism in FedPer and similar algorithms.

• Function: Makes final predictions (e.g., classification) based on features extracted by shared global layers.
• Rationale: The optimal decision boundary often varies per client due to local data distribution shifts.
• Example: In a federated image classifier for mobile phones, the shared backbone learns general features like edges and textures, while each user's local head learns to recognize their specific pets or household objects.

A design strategy for partitioning a neural network into global (federated) and local (personalized) components. FedPer is a canonical example.

• Architectural Choice: Engineers decide which layers are aggregated and which remain client-specific.
• Typical Split: Early layers (feature extractors) are global; later layers (classifiers/regressors) are personal.
• Granularity Control: Allows balancing between shared knowledge and local adaptation. More personalized layers increase flexibility but reduce collaborative learning benefits.

A closely related PFL algorithm that, like FedPer, decouples representation learning from task-specific adaptation.

• Mechanism: Learns a global feature representation (body) across all clients. Each client trains a unique local head (classifier) that operates on these shared features.
• Key Difference from FedPer: FedRep often employs an alternating training strategy: multiple local steps to learn the head, then a federated round to update the representation. FedPer typically trains both concurrently but only aggregates the base layers.
• Use Case: Effective when the feature space is common but label distributions are client-specific.

A straightforward PFL technique where a global model is downloaded and then further trained on a client's local data.

• Process: The client performs additional epochs of gradient descent using its private dataset.
• Contrast with FedPer: Fine-tuning adjusts all model parameters locally. FedPer keeps the personalization layer structure fixed from the start and never aggregates those layers.
• Trade-off: Fine-tuning is simple but can cause catastrophic forgetting of useful global knowledge. FedPer's architectural separation provides a more stable inductive bias for personalization.

A perspective framing PFL as transferring a global "hypothesis" (model) to local tasks.

• Concept: The global federated model serves as a strong source model or prior. Local training adapts this prior to the target (client) distribution.
• Connection to FedPer: FedPer's global base layers constitute a transferred feature extraction hypothesis. The local head learns a new hypothesis for the output function.
• Regularization: Often used alongside, adding a term (e.g., L2) to the local loss to penalize deviation from the global model, preventing the local model from drifting too far from useful shared knowledge.