FedRep

FedRep's core innovation is its architectural separation of the neural network into two distinct, independently learned components:

• Global Representation (θ): The shared feature extractor (e.g., convolutional layers). This is trained collaboratively across all clients to learn a common, useful feature space.
• Local Heads (wᵢ): The client-specific classifiers or regressors (e.g., final fully-connected layers). These are trained exclusively on each client's local data and are never shared or aggregated. This decoupling allows the model to capture general patterns globally while adapting decision boundaries locally.

FedRep operates via an alternating, two-phase training procedure within each communication round:

1. Local Head Update Phase: Clients freeze the global representation θ and perform multiple steps of stochastic gradient descent (SGD) only on their local head parameters wᵢ. This adapts the classifier to the client's unique data distribution.
2. Global Representation Update Phase: Clients freeze their now-personalized local heads wᵢ and perform multiple SGD steps only on the global representation θ. The gradients for θ are computed using the local data and the fixed, personalized head, ensuring the learned features are useful for each client's specific task. The server then aggregates the updated θ parameters from participating clients using Federated Averaging (FedAvg).

Personalization is not an afterthought but the direct objective of the training loop. The local head wᵢ is the personalized component. Because it is trained solely on client i's data and never averaged, it encodes the client-specific decision logic. At inference time, a client uses the combination of the latest global representation θ and its own persistently stored local head wᵢ. This provides a model tailored to the client's data distribution without exposing the raw local data.

FedRep inherently combats the negative effects of client drift—where local models diverge due to non-IID data. By alternating the update of θ and wᵢ, it creates a stabilizing effect:

• Updating wᵢ with a fixed θ allows the head to specialize without corrupting the shared feature space.
• Updating θ with a fixed, personalized wᵢ ensures the feature extractor learns representations that are generically useful for the current personalized heads, aligning the global update direction with local objectives. This contrasts with FedAvg, where all parameters drift, often harming convergence on heterogeneous data.

FedRep is often discussed alongside FedPer. Both are layer-wise personalization algorithms, but they differ in training methodology:

• FedPer: Also keeps base layers global and the head local. However, in a communication round, clients typically update all parameters (base and head) simultaneously. Only the base layers are sent to the server for aggregation.
• FedRep: Employs the explicit bi-phased update, alternating between head-only and representation-only training. This structured alternation is designed to more cleanly separate the learning objectives. Empirically, FedRep often demonstrates faster convergence and better final accuracy on highly non-IID data compared to FedPer.

Ideal For:

• Scenarios with statistical heterogeneity (non-IID data) across clients, such as different writing styles in next-word prediction or regional variations in medical imaging.
• Applications where a shared feature space is logical (e.g., visual features in images, semantic embeddings in text) but decision boundaries vary per client.

Limitations & Considerations:

• Architectural Constraint: Requires a clear delineation between 'representation' and 'head' layers, which may not suit all model architectures.
• Communication Cost: Similar to FedAvg, as the global representation θ is transmitted each round.
• Head Initialization: The performance can be sensitive to the initialization of the local heads for new clients joining the federation.

The overarching paradigm where a global model is trained collaboratively across clients, but the final deployed model is adapted to each client's unique data distribution. PFL addresses statistical heterogeneity (non-IID data) by moving beyond a single global model. Core challenges include balancing personalization with shared knowledge and managing communication costs.

• Goal: Learn a set of models {θ₁, θ₂, ..., θₙ} for N clients.
• Contrast with Standard FL: Standard FL aims for a single global model θᴳ; PFL aims for N personalized models.
• Applications: Next-word prediction on personal devices, healthcare diagnostics per hospital, retail recommendations per store.

A foundational PFL algorithm that architecturally separates a model into base layers (global, federated) and personalized layers (local, not aggregated). FedRep is a direct conceptual descendant of FedPer.

• Mechanism: The convolutional or feature-extracting base is learned collaboratively. The final classification head stays on-device.
• Key Difference from FedRep: FedPer personalizes only the final task-specific layers. FedRep explicitly learns a global representation across all clients and unique local heads, providing a more structured decoupling.
• Use Case: Standard benchmark for layer-wise personalization strategies.

Specific components of a neural network designated to be client-specific. In FedRep, the local head is the personalization layer, while the representation network is global.

• Typical Location: The final layers of a model (e.g., classifier head).
• Design Choice: The choice of which layers to personalize is a hyperparameter. Earlier layers capture general features; later layers capture task-specific details.
• Impact: Reduces communication payload (only global layers are transmitted) and protects client-specific patterns from being averaged away.

A straightforward PFL technique where a client receives the global model and performs additional training steps exclusively on its local data. This is often used as a baseline or post-processing step.

• Process: θᵢ = FineTune(θᴳ, Dᵢ) for client i with data Dᵢ.
• Relation to FedRep: FedRep can be seen as a more integrated approach where local head training and global representation learning are co-optimized during federation, rather than fine-tuning as a separate phase.
• Risk: Can lead to client drift if unregularized, where models diverge excessively from useful shared knowledge.

A PFL-adjacent approach where clients are partitioned into clusters based on data distribution similarity. A separate global model is learned for each cluster.

• Contrast with FedRep: FedRep creates N personalized models. Clustered FL creates K model (where K << N). FedRep's global representation can be seen as a single, rich feature space serving all clusters.
• Mechanism: Uses client update similarity or data statistics to form clusters. Useful when clear data modalities exist (e.g., image classifiers for dogs vs. cats across clients).
• Benefit: Reduces personalization overhead by sharing models within similar groups.

A perspective framing PFL as transferring the global model (the source hypothesis) to each client's local task, often with regularization to prevent catastrophic forgetting of useful shared knowledge.

• Mathematical Form: Local loss often includes a term like λ ||θᵢ - θᴳ||², penalizing deviation from the global model.
• Relation to FedRep: FedRep's structure inherently facilitates transfer. The global representation is the transferred hypothesis for feature extraction, which the local head then specializes.
• Advantage: Provides a theoretical framework for analyzing the stability and convergence of PFL algorithms.