Federated Hyperparameter Optimization

The simplest form of distributed HPO where a predefined set of hyperparameter combinations is evaluated in parallel across cross-silo clients. Each client trains a local model with an assigned configuration, and the global validation loss is aggregated to select the winner. While embarrassingly parallel, this approach suffers from the curse of dimensionality and is communication-inefficient, as every trial runs for the full number of local epochs regardless of intermediate performance.

An evolutionary HPO method adapted for federated learning where a population of models is trained in parallel on different client silos. Models periodically share their weights and hyperparameters with a central orchestrator. Underperforming agents are replaced by mutated versions of top performers through exploit-and-explore cycles. This allows hyperparameters to adapt dynamically during training, not just between runs, making it suitable for non-stationary clinical data distributions.

Techniques like Successive Halving or Hyperband adapted to a federated context. Instead of training every configuration to completion, a large number of configurations are evaluated on a low-fidelity proxy, such as a reduced number of local epochs or a subset of participating clients. Only the most promising configurations are promoted to higher fidelities. This drastically reduces the aggregate compute and communication budget required to find optimal hyperparameters.

A sequential model-based optimization approach that builds a surrogate probabilistic model (e.g., a Gaussian Process) of the validation loss as a function of hyperparameters. An acquisition function intelligently suggests the next configuration to evaluate, balancing exploration and exploitation. In the federated setting, the surrogate must be updated without centralizing raw performance metrics, often using secure aggregation of loss values to maintain privacy guarantees.

An extension of HPO that searches for the optimal model architecture itself, not just training hyperparameters. In a federated context, a controller proposes candidate architectures that are dispatched to cross-silo clients for local training and evaluation. The aggregated performance metrics guide the controller's search strategy, often using reinforcement learning or evolutionary algorithms, to discover architectures that generalize well across heterogeneous clinical data distributions.

A critical consideration where the hyperparameter search itself can leak sensitive information. For example, the optimal learning rate might reveal properties of the private training data. DP-HPO frameworks add calibrated noise to the validation metrics reported by each client before aggregation, ensuring that the selected hyperparameters provide a (ε, δ)-differential privacy guarantee. This consumes a portion of the overall privacy budget.

The phenomenon where local models trained on heterogeneous, non-IID data diverge from each other and the optimal global model. Hyperparameter optimization must account for client drift by tuning proximal terms in the loss function, such as the mu parameter in FedProx, or by adjusting the number of local training steps. Excessive local training on skewed data distributions exacerbates drift, leading to slow convergence or degraded global model performance.

A quantitative metric used to measure the degree of statistical heterogeneity across decentralized datasets, often calculated using Earth Mover's Distance or Dirichlet distribution parameters. This index serves as a critical input to hyperparameter optimization strategies, as the optimal learning rate and aggregation frequency are highly dependent on the degree of label distribution skew and feature distribution skew across participating clinical sites.

Techniques for minimizing bandwidth overhead in decentralized training, including gradient compression, quantization, and sparsification. Hyperparameter optimization in this context involves tuning the compression ratio and the frequency of communication rounds. For example, a higher compression ratio reduces bandwidth but may require more local epochs to compensate for the information loss, creating a trade-off space that must be systematically explored.

A model selection technique adapted for decentralized data where the partitioning of data into folds respects institutional boundaries. Hyperparameter optimization relies on federated cross-validation to evaluate different configurations without centralizing validation data. The key constraint is that a client's data is never split between training and validation sets, ensuring that institutional data sovereignty is maintained throughout the tuning process.

A mathematical framework that provides a quantifiable guarantee that the output of a computation reveals no information about any single individual's data. Hyperparameter optimization in differentially private federated learning involves tuning the noise multiplier and clipping norm alongside traditional parameters. The privacy budget (epsilon) acts as a hard constraint that directly limits the number of hyperparameter evaluation trials possible before the budget is exhausted.