Bayesian Optimization

The surrogate model is a probabilistic approximation of the expensive, black-box objective function. It is trained on all previously evaluated points (hyperparameter sets and their performance scores).

• Gaussian Process (GP) is the most common choice, as it provides a full posterior distribution (mean and variance) for any unseen point, quantifying prediction uncertainty.
• The model's posterior mean predicts performance, while its posterior variance represents uncertainty about that prediction.
• This model is cheap to query, allowing the algorithm to reason about millions of potential configurations without running the actual costly training job.

The acquisition function is a utility function that uses the surrogate model's predictions to decide which hyperparameter set to evaluate next. It automatically balances exploration (probing uncertain regions) and exploitation (refining known good regions).

Common acquisition functions include:

• Expected Improvement (EI): Measures the expected improvement over the current best observation.
• Upper Confidence Bound (UCB): Selects points with a high upper confidence bound (mean + κ * variance).
• Probability of Improvement (PI): Measures the probability that a point will be better than the current best. The next point to evaluate is chosen by maximizing the acquisition function, a much cheaper optimization problem.

Bayesian optimization operates via a closed-loop, sequential process:

1. Initialize: Evaluate the objective function at a few random points to build an initial surrogate model.
2. Fit Surrogate: Train the probabilistic model (e.g., Gaussian Process) on all observed data.
3. Optimize Acquisition: Find the hyperparameters that maximize the acquisition function.
4. Evaluate & Update: Run the expensive evaluation (e.g., train the model) at the proposed point. Add the new (hyperparameters, score) pair to the observation history.
5. Repeat: Loop back to step 2 until a budget (e.g., number of trials) is exhausted. This loop ensures each evaluation is chosen to provide maximum information about the location of the optimum.

The Gaussian Process (GP) prior defines assumptions about the smoothness and structure of the objective function. It is specified by a mean function (often zero) and a kernel/covariance function.

• The kernel (e.g., Matérn, Radial Basis Function) controls how correlation between points decays with distance, modeling function smoothness.
• Hyperparameters of the GP itself (like length-scale) can be optimized by maximizing the marginal likelihood of the observed data.
• This prior allows the model to generalize from observed points and provide well-calibrated uncertainty estimates, which is critical for the acquisition function's exploration-exploitation trade-off.

In Parameter-Efficient Fine-Tuning (PEFT), Bayesian optimization is used to find optimal configurations for adaptation methods, which have complex, non-convex hyperparameter landscapes.

Example search spaces include:

• LoRA: Rank (r), alpha scaling, dropout rate, target modules.
• Adapter Networks: Bottleneck dimension, reduction factor, placement location.
• Prompt/Prefix Tuning: Length of continuous prompts, initialization strategy. Each evaluation involves fine-tuning the large base model with the PEFT method, making BO's sample efficiency essential for cost-effective adaptation.

Bayesian optimization is a foundational technique within the broader ecosystem of automated machine learning (AutoML) and neural configuration:

• Hyperparameter Optimization (HPO): BO is a premier global HPO method, superior to grid/random search for expensive functions.
• Neural Architecture Search (NAS): BO can search over discrete architectural choices (e.g., number of layers, operation types) by using a surrogate model over the graph space.
• Multi-Objective BO: Extends BO to optimize for competing goals (e.g., accuracy vs. model size), finding a Pareto frontier of optimal trade-offs.
• Hypernetworks: BO can tune the hyperparameters of a hypernetwork that generates weights for a main PEFT module.

Hyperparameter optimization (HPO) is the broader, automated process of searching for the optimal set of configuration parameters that govern a model's learning process (e.g., learning rate, batch size, regularization strength).

• Goal: Maximize model performance (accuracy, F1-score) on a validation set.
• Scope: Encompasses both black-box methods like Bayesian optimization and gradient-based methods.
• Key Distinction: While Bayesian optimization is a specific strategy for HPO, HPO itself is the overarching objective.

A surrogate model is a probabilistic, computationally inexpensive model trained to approximate the behavior of a costly-to-evaluate objective function, such as a model's validation loss.

• Core Component: The heart of the Bayesian optimization loop.
• Common Choice: Gaussian Processes (GPs) are frequently used for their native uncertainty estimates.
• Function: It models the posterior distribution of the objective, predicting both the expected performance and the uncertainty for unseen hyperparameter configurations, guiding the acquisition function.

The acquisition function is a utility function that uses the surrogate model's predictions to decide which hyperparameter set to evaluate next in the Bayesian optimization loop.

• Purpose: Balances exploration (testing uncertain regions) and exploitation (refining known good regions).
• Common Types:
  • Expected Improvement (EI): Measures the expected improvement over the current best observation.
  • Upper Confidence Bound (UCB): Optimistically selects points with high predicted mean plus a weighted uncertainty term.
  • Probability of Improvement (PI): Selects points most likely to be better than the current best.

Neural architecture search (NAS) is a subfield of AutoML focused on algorithmically discovering high-performing neural network architectures. Bayesian optimization is a foundational strategy for guiding this search.

• Search Space: Defines possible layer types, connections, and operations.
• Objective: Often a multi-objective trade-off between accuracy, latency, and model size.
• Connection: In NAS, the "black-box function" is the process of training and evaluating a candidate architecture. Bayesian optimization provides a sample-efficient framework for navigating this vast, discrete search space.

Automated machine learning (AutoML) aims to automate the end-to-end process of applying ML to real-world problems. Bayesian optimization is a critical enabling technology within AutoML pipelines.

• Broader Scope: Beyond HPO/NAS, AutoML can include automated feature engineering, model selection, and pipeline creation.
• Role of BO: Serves as the optimization engine for tuning hyperparameters across any stage of the pipeline where performance is a black-box function of configuration choices.
• Example Systems: Platforms like Google's Vertex AI and Auto-sklearn use Bayesian optimization at their core.

Multi-objective optimization extends Bayesian optimization to problems with several, often competing, goals—such as maximizing accuracy while minimizing model size and inference latency.

• Output: A Pareto front, representing the set of optimal trade-off solutions.
• Acquisition Adaptation: Functions like Expected Hypervolume Improvement (EHVI) are used to evaluate candidates based on their contribution to improving the Pareto front.
• Critical for PEFT/Edge AI: Directly relevant for configuring parameter-efficient methods where the trade-off between adaptation quality and parameter/compute budget is paramount.