Bayesian Optimization

The surrogate model is a probabilistic, computationally inexpensive model trained to approximate the expensive black-box objective function. It quantifies prediction uncertainty.

• Primary Function: Maps hyperparameters to a distribution over possible performance scores (e.g., validation loss).
• Common Choice: Gaussian Process (GP) is favored for its natural uncertainty estimates. Tree-based models like Random Forests are also used.
• Output: For a given input x (hyperparameters), it provides a mean prediction μ(x) and an uncertainty estimate σ(x).

The acquisition function is a criterion that uses the surrogate model's predictions to decide the next hyperparameter set to evaluate by balancing exploration and exploitation.

• Purpose: Automates the decision of "where to sample next."
• Common Functions:
  • Expected Improvement (EI): Measures the expected gain over the current best observation.
  • Upper Confidence Bound (UCB): Selects points with high μ(x) + κ * σ(x), where κ controls the exploration-exploitation trade-off.
  • Probability of Improvement (PI): Measures the probability that a new point will be better than the current best.
• Optimization: The acquisition function itself is cheap to evaluate, allowing it to be optimized (e.g., via gradient descent) to propose the next candidate.

The observation history is the set of previously evaluated {hyperparameters, objective value} pairs. It is the empirical data that fuels the sequential optimization loop.

• Initialization: Typically begins with a small set of random points or points from a space-filling design (e.g., Latin Hypercube Sampling).
• Iterative Growth: With each iteration (t), a new point x_t is evaluated on the true expensive function, and the pair (x_t, y_t) is added to the history.
• Role: This growing dataset is used to retrain/update the surrogate model after each iteration, improving its approximation of the objective landscape.

The optimization loop is the sequential, closed-loop process that integrates all components. It typically runs for a fixed budget of evaluations.

Standard Loop (Iteration t):

1. Fit Surrogate: Train the probabilistic model (e.g., Gaussian Process) on the current observation history.
2. Maximize Acquisition: Find the hyperparameters x_t that maximize the acquisition function α(x).
3. Evaluate Objective: Compute the expensive objective function y_t = f(x_t).
4. Update History: Augment the dataset with (x_t, y_t).

This loop continues until the evaluation budget (e.g., 100 trials) is exhausted. The final output is the best configuration found in the history.

When using a Gaussian Process as the surrogate, the kernel function is its core component, defining the similarity or covariance between data points.

• Purpose: Encodes assumptions about the smoothness and periodicity of the objective function.
• Common Kernels:
  • Matérn Kernel: A flexible default, with parameter ν controlling smoothness (ν=5/2 is common).
  • Squared Exponential (RBF): Assumes very smooth functions.
• Hyperparameters: Kernels have their own parameters (e.g., length-scales) that are optimized during GP fitting, automatically adapting to the scale of each hyperparameter.

Bayesian Optimization is one strategy within the broader family of Sequential Model-Based Optimization (SMBO). Key related approaches include:

• Multi-Fidelity Optimization: Uses cheaper, low-fidelity approximations (e.g., training for fewer epochs) to guide the search, as seen in Hyperband.
• Population-Based Training (PBT): A hybrid method that evolves a population of models, combining weight inheritance with hyperparameter perturbation.
• Gradient-Based Optimization: Treats hyperparameters as differentiable and computes gradients via implicit differentiation (conceptually different, as BO is derivative-free).

BO is distinguished by its explicit probabilistic modeling and use of an acquisition function for decision-making under uncertainty.

A surrogate model is a computationally inexpensive probabilistic model trained to approximate the relationship between hyperparameter configurations and a target objective (like validation loss). In Bayesian Optimization, it replaces the expensive evaluation of the true black-box function. Common choices include:

• Gaussian Processes (GPs): Provide uncertainty estimates crucial for the exploration-exploitation trade-off.
• Tree-structured Parzen Estimators (TPE): Often used in high-dimensional or conditional spaces.
• Random Forests: Scale well and handle categorical parameters. The surrogate is updated after each evaluation to refine its predictions and guide the search.

An acquisition function is the decision-making engine of Bayesian Optimization. After the surrogate model provides a probabilistic prediction of performance across the search space, the acquisition function calculates a score for each point, balancing:

• Exploration: Sampling from regions of high uncertainty.
• Exploitation: Sampling near the current best-known optimum. The next hyperparameter set to evaluate is the one that maximizes this function. Common acquisition functions include:
• Expected Improvement (EI): Measures the expected gain over the current best.
• Upper Confidence Bound (UCB): Adds a confidence interval to the predicted mean.
• Probability of Improvement (PI): Simpler, but can be less exploratory.

Hyperparameter Optimization (HPO) is the broader problem that Bayesian Optimization solves. It is the automated search for the optimal set of hyperparameters—configuration settings that govern a model's training process (e.g., learning rate, batch size, regularization strength). Unlike model parameters learned during training, hyperparameters are set beforehand. HPO strategies include:

• Grid Search: Exhaustive but computationally prohibitive.
• Random Search: More efficient baseline.
• Bayesian Optimization: The model-based, sequential approach that is most sample-efficient for expensive-to-evaluate functions. The goal is to maximize a model's performance on a held-out validation set.

Multi-fidelity optimization is a strategy to accelerate Bayesian Optimization and other HPO methods by leveraging cheaper, lower-fidelity approximations of model performance. Instead of always training a model to convergence, it uses proxies like:

• Training on a subset of data.
• Training for a reduced number of epochs.
• Using a lower-resolution model. Algorithms like Hyperband and BOHB (Bayesian Optimization and HyperBand) use this principle. They quickly discard poor configurations using low-fidelity evaluations and allocate full resources only to the most promising candidates, dramatically improving search efficiency.

Neural Architecture Search (NAS) is a subfield of AutoML focused on automating the design of neural network architectures. Bayesian Optimization can be applied to NAS by treating architectural choices (e.g., number of layers, types of operations) as a complex, high-dimensional hyperparameter search. More advanced NAS methods include:

• Differentiable NAS (DARTS): Uses gradient-based optimization within a continuous relaxation of the architecture space.
• Evolutionary NAS: Uses evolutionary algorithms for population-based search.
• One-Shot NAS: Trains a single supernet encompassing all architectures, with weight sharing for efficient evaluation. The objective is to discover task-specific architectures that outperform human-designed counterparts.

Model-Agnostic Meta-Learning (MAML) is a gradient-based meta-learning algorithm that learns a model initialization which can rapidly adapt to new tasks with few gradient steps. While distinct from Bayesian Optimization, both address the challenge of optimizing in sample-efficient ways. MAML's "learning to learn" paradigm is complementary:

• Bayesian Optimization efficiently optimizes hyperparameters for a single task.
• MAML finds an initialization that enables fast adaptation to many related tasks. In advanced AutoML systems, meta-learned priors can be used to warm-start a Bayesian Optimization procedure, leveraging knowledge from previous tasks to accelerate convergence on a new one.