Federated Hyperparameter Optimization

Federated Bayesian Optimization is the predominant method for FedHPO. It constructs a global surrogate model (typically a Gaussian Process) of the hyperparameter-performance landscape by aggregating observations from clients.

• Mechanism: Clients evaluate hyperparameter configurations locally and report performance metrics (e.g., validation loss) to the server. The server updates the surrogate model and uses an acquisition function (like Expected Improvement) to select the next promising configuration to test.
• Privacy: Only scalar performance metrics are shared, not raw data or gradients.
• Challenge: The surrogate model must account for client heterogeneity; a configuration performing well on average may fail on specific client distributions.

Population-Based Training adapts evolutionary algorithms for federated settings. A population of models with different hyperparameters is maintained and evolved across clients.

• Mechanism: Each client trains a member of the population. Periodically, the server performs selection (e.g., based on client-reported fitness) and generates new hyperparameter sets via mutation and crossover. Promising configurations replace poor ones.
• Advantage: Simultaneously optimizes hyperparameters and model weights, and can adapt hyperparameters during training.
• Use Case: Effective for tuning adaptive learning rate schedules and regularization parameters in non-stationary environments.

This method estimates gradients of the validation loss with respect to hyperparameters (hypergradients) in a federated manner.

• Mechanism: Clients compute implicit gradients or approximations of how hyperparameters affect their local validation loss. These local hypergradients are aggregated at the server to perform gradient-based updates on the hyperparameters themselves.
• Example: Used to federate the tuning of client learning rates by differentiating through the local SGD steps.
• Limitation: Computationally intensive and requires careful design to avoid exposing client data through the gradient computation.

This communication-efficient method applies early-stopping principles across clients. It allocates more resources only to promising hyperparameter configurations.

• Mechanism: Configurations are evaluated for a few local epochs on a subset of clients. The worst-performing half are discarded (successive halving). The remaining configurations are evaluated for more epochs and/or on more clients in the next round.
• Benefit: Dramatically reduces total client compute and communication by avoiding full training runs on poor hyperparameters.
• Adaptation: Federated Hyperband is a common instantiation that runs Successive Halving with multiple resource budgets in parallel.

This approach recognizes that optimal hyperparameters may differ per client due to data heterogeneity. It aims to find a set of hyperparameters or a strategy that yields good personalized models.

• Methods:
  • Meta-Learning: Learn a global hyperparameter initialization that allows for fast local adaptation (few-shot HPO) on each client.
  • Contextual BO: The surrogate model conditions hyperparameter recommendations on client context (e.g., data distribution statistics).
  • Mixture of Experts: Train different global hyperparameter "experts" and learn a router to assign clients.
• Goal: Move beyond a single global optimum to a Pareto-optimal set for the federated population.

This method explicitly optimizes hyperparameters for the system-level objectives of a federated deployment, not just model accuracy.

• Optimized Metrics: Jointly tunes hyperparameters to balance:
  • Model Performance (e.g., global accuracy)
  • Resource Efficiency (e.g., total training time, communication rounds)
  • Fairness (e.g., performance disparity across clients)
  • Privacy Cost (e.g., the epsilon spent in differentially private training)
• Technique: Often formulated as a multi-objective optimization problem, solved using methods like federated multi-objective Bayesian optimization.
• Outcome: Produces configurations that are pragmatic for real-world FL system constraints.

The fundamental tension in Federated Hyperparameter Optimization (FedHPO) is between exploration (trying diverse hyperparameters to find the best configuration) and privacy preservation (avoiding data leakage through repeated queries to clients).

• Direct Evaluation Risk: Naively testing hyperparameters by training on client data risks exposing information through the model updates or the performance metrics themselves.
• Solution - Federated Proxy Metrics: Algorithms use federated validation on held-out client data or train surrogate models (like Gaussian Processes) on aggregated, anonymized performance statistics to guide the search without centralizing raw data.

Clients vary in hardware (compute, memory), connectivity (bandwidth, latency), and data distribution (non-IID). This heterogeneity makes consistent hyperparameter evaluation unreliable.

• Challenge: A learning rate optimal for a fast, well-connected client with balanced data may cause divergence for a slower client with skewed data.
• Solution - Asynchronous & Personalized HPO: Methods like Asynchronous Successive Halving (ASHA) allow clients to report results at different times. Personalized HPO strategies can recommend different hyperparameters per client or client cluster based on their resource and data profiles.

Traditional HPO requires many training trials. In federated learning, each trial corresponds to at least one full federated round, making exhaustive search prohibitively expensive.

• Cost Multiplier: Searching over 50 hyperparameter configurations with 100 communication rounds each results in 5,000 total federated rounds.
• Solution - Population-Based & One-Shot Methods: Population-Based Training (PBT) evolves hyperparameters online during a single training run. One-Shot Federated HPO uses weight-sharing architectures (like supernets) to evaluate many configurations in parallel within one round, drastically reducing communication.

Bayesian Optimization (BO) is a leading model-based approach for FedHPO. It builds a probabilistic surrogate model of the global objective function (validation loss) and uses an acquisition function to select the most promising hyperparameters to test next.

• Federated Adaptation: In Federated BO, the surrogate model is trained on aggregated, privacy-protected performance metrics from clients. The acquisition function must account for client heterogeneity.
• Example: A method might use Federated Thompson Sampling, where each client samples from the global surrogate model to decide on a local hyperparameter configuration, balancing exploration and exploitation.

These strategies are favored for their efficiency and robustness in decentralized, noisy environments.

• Federated Population-Based Training (FedPBT): A population of models is trained in parallel. Periodically, poorly performing models' hyperparameters are replaced by mutations of better-performing ones, and their weights are partially copied. This happens via federated averaging for the weights and rules for hyperparameters.
• Multi-Armed Bandit (MAB) Formulations: Each hyperparameter configuration is an 'arm'. Bandit algorithms like Federated Successive Halving (FedSH) or Hyperband dynamically allocate more training rounds to promising configurations while early-dropping poor ones, optimizing the communication budget.

The foundational aggregation algorithm for federated learning. FedAvg coordinates the core training loop where FedHPO operates:

• The server sends a global model to a subset of clients.
• Each client performs Local SGD for multiple epochs.
• Clients send their updated model weights back to the server.
• The server computes a weighted average of these updates to form a new global model. FedHPO tunes hyperparameters like the number of local epochs and client learning rate that directly govern this FedAvg process.

A primary challenge that FedHPO must mitigate. Client drift occurs when local models diverge significantly from the global objective due to:

• Non-IID Data: Statistically heterogeneous data distributions across clients.
• Excessive Local Epochs: Too many local SGD steps cause overfitting to the client's local data. FedHPO strategies, like Bayesian optimization, search for hyperparameter configurations (e.g., optimal local steps, personalized learning rates) that minimize this drift to ensure stable global convergence.

A class of server-side optimization algorithms that FedHPO can tune. Instead of simple weighted averaging (FedAvg), these methods apply adaptive optimizers to the aggregation step:

• FedAdam: Applies the Adam optimizer to client updates.
• FedYogi: A variant of Adam offering more stable updates.
• FedAdagrad: Applies per-parameter adaptive learning rates. FedHPO is used to find the optimal server learning rate, momentum parameters (β1, β2), and stabilization constant (ε) for these adaptive algorithms, which is crucial for performance on complex, non-convex models.

A closely related goal often co-optimized with FedHPO. The objective is to produce models tailored to individual client data distributions. FedHPO enables this by tuning:

• Personalized Learning Rates: Assigning different client-side learning rates.
• Regularization Strength: Controlling how much local models can deviate from the global model (e.g., via a proximal term as in FedProx).
• Mixture Weights: For algorithms that blend global and local models. Effective FedHPO finds the hyperparameter set that balances global model utility with strong local personalization.

A critical systems constraint that influences FedHPO design. The cost of communicating model updates drives hyperparameter choices:

• Local Epochs: More local computation reduces communication rounds but risks client drift.
• Client Participation Rate: Selecting more clients per round increases bandwidth use.
• Compression Techniques: FedHPO may tune parameters for methods like Gradient Compression or Quantized Gradient Communication. The hyperparameter search must optimize for final model accuracy within a total communication budget.

A formal privacy guarantee that introduces a key trade-off for FedHPO. Adding DP noise to client updates protects data but harms model utility. FedHPO must optimize hyperparameters that govern this trade-off:

• Noise Multiplier (σ): The standard deviation of the Gaussian noise added.
• Clipping Norm (C): The maximum L2 norm for client updates before adding noise.
• Sampling Rate (q): The probability of a client participating in a round. FedHPO searches for the configuration that achieves the target privacy budget (ε, δ) while maximizing final model accuracy.