Per-Client Hyperparameters

Per-client hyperparameters are unique configuration values assigned to individual clients in a federated learning system, governing their local training process. Unlike a single global setting, these parameters—such as learning rate, batch size, number of local epochs, and regularization strength—are tailored to each device's specific context.

• Mechanism: The server can either statically assign values based on client metadata (e.g., device class, connectivity) or dynamically learn them via meta-optimization.
• Purpose: This customization accounts for variations in local data distribution (statistical heterogeneity), computational capability, and network stability, preventing client drift and improving overall system efficiency.

The primary driver for per-client hyperparameters is statistical heterogeneity, where client data is not independently and identically distributed (Non-IID). A single global learning rate often fails in this setting.

• Problem: Clients with different data distributions converge to different local optima. A high global learning rate may cause divergent clients to overshoot, while a low rate slows convergence for well-aligned clients.
• Solution: Assigning a client-specific learning rate allows each device to take steps proportional to the alignment of its local gradient with the global objective. Techniques like Adaptive Federated Optimization (FedOpt) can implicitly create per-client learning dynamics through server-side adaptive optimizers.

Clients vary widely in hardware (CPU/GPU/Memory), power constraints, and network connectivity. Per-client hyperparameters adapt the training workload to these constraints.

• Local Epochs (E): A powerful, plugged-in device might use a higher E (e.g., 10) to perform more local computation, while a battery-sensitive mobile device uses a lower E (e.g., 2).
• Batch Size (B): Adjusted based on available device memory.
• Communication Frequency: Implicitly controlled by E and B, managing trade-offs between local computation and communication rounds. This prevents stragglers from bottlenecking the federation and enables participation from a more diverse device pool.

Key hyperparameters that are frequently personalized and their effects on training:

• Learning Rate (η): The most critical. A per-client η directly controls step size during local SGD, mitigating drift. Can be tuned via hyperparameter transfer or meta-learning.
• Number of Local Epochs (E): Determines how much local computation occurs per round. Higher E increases client drift but utilizes more local data.
• Regularization Coefficient (λ): Strength of terms like L2 regularization or proximal term (as in FedProx). A client-specific λ balances local model adaptation with proximity to the global model.
• Momentum Coefficient: Can be personalized to stabilize local optimization trajectories.

There are two primary methods for implementing per-client hyperparameters:

1. Static Assignment: The server maintains a lookup table or uses a heuristic function to assign values based on reported client metadata (e.g., device_type, avg_data_size). Simple but not adaptive.
2. Dynamic Learning (Meta-Learning): The server meta-learns a hyperparameter initialization or a policy. For example, using Reptile or MAML to find initial hyperparameters that allow fast adaptation. The server can also learn a hypernetwork that generates client-specific parameters from a client embedding.
3. Adaptive Server Optimizers: Using optimizers like FedAdam or FedYogi on the server applies adaptive, client-aware weighting to updates, which functionally creates personalized optimization dynamics.

Per-client hyperparameters are a complementary technique within the Personalized Federated Learning (PFL) toolbox, often combined with other methods:

• With Personalization Layers: Hyperparameters can control the training of both shared and local layers. For instance, a higher learning rate might be applied to a client's personalized model head.
• With Clustered FL: Hyperparameters can be assigned per cluster of similar clients, rather than per individual client, for scalability.
• With Regularization Methods: Techniques like FedProx's μ parameter, which controls attachment to the global model, can be made client-specific. The goal is synergistic: architectural personalization (e.g., FedPer) handles structural adaptation, while hyperparameter personalization optimizes the training process itself.

In a federated learning system for diabetic retinopathy detection across hospitals, per-client learning rates are essential. A hospital with a large, high-quality dataset of annotated fundus images might use a standard learning rate (e.g., 0.001). In contrast, a clinic with a smaller, noisier dataset uses a lower, more conservative learning rate (e.g., 0.0001) to prevent overfitting to local artifacts. This adaptive tuning ensures stable convergence for all participants without requiring data sharing.

A global bank trains a fraud detection model across regional branches. Transaction patterns differ drastically:

• Region A (High-volume e-commerce): Uses stronger L2 regularization to prevent the model from overfitting to prevalent but benign online shopping patterns.
• Region B (Emerging market, cash-heavy): Applies weaker regularization to allow the model to be more sensitive to subtle, novel fraud signatures in a sparse data environment. This per-client regularization strength acts as a complexity controller tailored to each branch's data distribution and risk profile.

In a network of industrial vibration sensors for predictive maintenance, device capabilities vary. Per-client local epochs are tuned dynamically:

• High-power gateway sensor: Can perform 10 local training epochs per federated round for thorough learning.
• Battery-powered edge sensor: Restricted to 2 local epochs to conserve energy. The server orchestrates this by sending client-specific local_epochs hyperparameters, ensuring participation from heterogeneous hardware without draining constrained devices.

For a federated keyboard app learning next-word prediction, user typing styles are highly personal. The system employs per-client optimizer momentum:

• User with formal, consistent writing: Benefits from high momentum (e.g., 0.9), allowing the local optimizer to build velocity in the consistent gradient direction of their prose.
• User with eclectic, code-mixed slang: Uses lower momentum (e.g., 0.5), making the optimizer more agile and responsive to abrupt changes in linguistic context. This fine-tuning at the optimizer level personalizes the training dynamics for each user's unique data stream.

An autonomous vehicle manufacturer improves a perception model using data from its global fleet. Per-client batch size is critical due to varying onboard compute:

• Latest model sedan with a powerful GPU: Uses a large batch size (e.g., 64) for stable gradient estimates.
• Older model economy car with an embedded processor: Uses a small batch size (e.g., 8) to fit training into limited memory. The server manages this heterogeneity, ensuring all vehicles contribute effectively to the global model while respecting their hardware constraints.

Advanced systems don't just assign hyperparameters—they learn them. Techniques include:

• Hypernetworks: A small neural network on the server takes client metadata as input and outputs that client's optimal hyperparameters.
• Bayesian Optimization: The server treats each client's performance as a black-box function and uses a surrogate model to propose better hyperparameters over successive federated rounds.
• Gradient-Based Meta-Learning: Algorithms like MAML are adapted to meta-learn hyperparameter initialization that is easily adaptable to new clients. This automates the personalization of the personalization process itself.

Personalized Federated Averaging (pFedAvg) is the foundational algorithm that extends standard FedAvg to produce client-specific models. Its core innovation is the decoupling of model parameters: a global shared component is learned via federated aggregation, while a local personalized component is updated only on the client's own data. This allows each client to maintain a model uniquely adapted to its local data distribution without exposing its private parameters to the server.

• Key Mechanism: Alternates between federated rounds for global parameters and purely local updates for personalization layers.
• Example: In a next-word prediction task, pFedAvg could learn a global language representation while allowing each user's phone to fine-tune the final output layer on their personal typing history.

FedPer and FedRep are two canonical algorithms that implement layer-wise personalization, a design strategy critical for managing per-client hyperparameters.

• FedPer: Keeps the base layers (feature extractor) of a neural network global and federated, while the classification head (personalized layers) remains local and is never aggregated. This is ideal for tasks where data heterogeneity is primarily in the label distribution.
• FedRep: Takes a two-phase approach. It learns a global representation across all clients, then freezes it to learn unique local heads for each client. This decouples shared feature learning from client-specific decision boundaries.

Both architectures provide a clear separation between shared and personalizable parameters, forming a structural basis for applying distinct hyperparameters to each component.

Local fine-tuning is the most straightforward PFL technique and a direct consumer of per-client hyperparameters. After receiving a global model from the federated server, each client performs additional training steps exclusively on its local dataset. The client-specific learning rate, number of epochs, and optimizer choice are classic examples of per-client hyperparameters tuned during this phase.

• Purpose: Adapts the general global model to the client's specific data patterns, improving local accuracy.
• Risk: Uncontrolled fine-tuning can lead to catastrophic forgetting of valuable global knowledge or client drift, where models diverge excessively. Techniques like regularized local training (e.g., adding a term penalizing distance from the global model) are used to mitigate this.

Personalized Federated Meta-Learning (PFML) frames PFL as a meta-learning problem. The goal is to learn a global model initialization that is explicitly optimized for fast adaptation. In the inner loop, clients perform a few steps of local training (fine-tuning). In the outer loop, the server updates the global initialization based on how well the adapted models perform on local validation sets.

• Connection to Hyperparameters: The meta-learning process inherently learns adaptation strategies that can subsume the need for manually tuning per-client hyperparameters like the local learning rate. The model learns how to personalize efficiently.
• Benefit: Particularly effective for new clients with very limited data, as the model is primed for rapid personalization.

A Mixture of Experts (MoE) architecture for PFL maintains a global set of specialized sub-models ("experts"). Each client has a lightweight, personalized gating network that determines how to combine these experts for local inference. Only the gating networks are client-specific, while the experts are trained collaboratively across all clients.

• Hyperparameter Role: The gating network's architecture and training (e.g., its learning rate) become key per-client hyperparameters. The sparsity factor (how many experts are activated per input) can also be tuned per-client for efficiency.
• Advantage: Enables sophisticated personalization by dynamically routing inputs to the most relevant experts based on local data patterns, without requiring a fully unique model per client.

FedBN is a simple yet powerful technique that addresses feature distribution shift—a major source of data heterogeneity. It modifies the standard federated training protocol by keeping the parameters of Batch Normalization (BN) layers local to each client. The client-specific mean and variance statistics are not aggregated with the server.

• Mechanism: During local training, BN layers use local statistics. During inference, they use the client's own stored statistics. All other model weights are federated normally.
• Significance: This is a form of implicit per-client hyperparameterization, where the BN parameters act as adaptive calibrators for local data. It demonstrates that personalization can be achieved by isolating specific, sensitive parameters rather than entire layers, offering a lightweight alternative to full local fine-tuning.