Bayesian Optimization

Instead of directly evaluating the expensive objective function, Bayesian Optimization builds a cheap-to-evaluate probabilistic proxy—typically a Gaussian Process (GP). This model predicts the objective value and quantifies the epistemic uncertainty at any unobserved point. As new evaluations are collected, the surrogate is updated via Bayesian inference, refining its posterior distribution over the true function.

The acquisition function is the mathematical heuristic that guides the search by assigning a utility score to candidate points. It explicitly balances the exploration-exploitation trade-off:

• Expected Improvement (EI): Measures the expected gain over the current best observation.
• Upper Confidence Bound (UCB): Optimistically combines predicted mean and variance.
• Probability of Improvement (PI): Maximizes the chance of exceeding the incumbent optimum. The point maximizing this function is selected for the next expensive evaluation.

Bayesian Optimization is the dominant paradigm when each function evaluation is extremely costly—financially, computationally, or temporally. Unlike grid search or random search, it reasons about where to sample next based on all prior observations. This makes it ideal for:

• Hyperparameter tuning of deep neural networks (e.g., learning rate, layer count) where a single training run costs thousands of GPU-hours.
• A/B testing where exposing a suboptimal variant to real users incurs revenue loss.
• Experimental design in wet-lab biology or chemistry where physical experiments are slow and expensive.

Bayesian Optimization operates as a closed-loop, iterative process:

1. Fit the surrogate model (e.g., Gaussian Process) to all historical observations.
2. Optimize the acquisition function to propose the next sampling location.
3. Evaluate the true objective function at the proposed point.
4. Augment the dataset with the new observation and repeat. This sequential design ensures that each step is informed by the maximum amount of accumulated evidence, converging to the global optimum with minimal evaluations.

The behavior of the Gaussian Process surrogate is governed by its covariance kernel. The kernel encodes prior structural assumptions about the objective function's smoothness and periodicity. Common choices include:

• Radial Basis Function (RBF): Assumes the function is infinitely differentiable and smooth.
• Matérn Kernel: A flexible family that models less smooth, more realistic physical processes.
• Periodic Kernel: Captures cyclical patterns in the data. Selecting the right kernel is critical for accurate uncertainty calibration and convergence speed.

Unlike gradient-based optimizers that can be trapped in local minima, Bayesian Optimization is a global black-box optimizer. By maintaining a full probabilistic model of the objective landscape, it naturally avoids premature convergence. The acquisition function explicitly incentivizes visiting regions with high predictive uncertainty, ensuring the optimizer does not overlook a distant, narrow global peak while exploiting a wide local one.

A probabilistic approximation of the expensive black-box objective function that is cheap to evaluate. In Bayesian optimization, the surrogate—typically a Gaussian Process (GP)—provides both a predicted mean and a quantified epistemic uncertainty at every point in the search space. The surrogate is updated with each new observation, progressively refining its belief about the true function's shape. Common alternatives to GPs include Random Forests and Tree-structured Parzen Estimators (TPE).

• Role: Replaces the true expensive function during the search.
• Key property: Must provide well-calibrated uncertainty estimates.
• Example: A GP surrogate modeling the validation accuracy of a deep neural network as a function of its hyperparameters.

A utility function that guides the search by quantifying the expected value of evaluating the true objective at any candidate point. The acquisition function balances the exploration-exploitation trade-off, proposing points where the surrogate model's uncertainty is high (exploration) or where the predicted objective value is promising (exploitation). The next point to evaluate is the one that maximizes the acquisition function.

• Expected Improvement (EI): Measures the expected positive gain over the current best observation.
• Probability of Improvement (PI): The probability that a point will improve upon the current best.
• Upper Confidence Bound (UCB): An optimistic estimate combining the predicted mean and variance.

A non-parametric, probabilistic model that defines a distribution over functions. A GP is fully specified by a mean function and a covariance function (kernel). The kernel encodes assumptions about the smoothness and structure of the objective function, such as the Matérn or Radial Basis Function (RBF) kernel. GPs are the dominant surrogate model in Bayesian optimization because they provide closed-form, analytically tractable posterior predictions with well-calibrated uncertainty.

• Advantage: Principled uncertainty quantification.
• Limitation: Computational complexity scales cubically, O(n³), with the number of observations, making it challenging for high-dimensional or large-sample problems.

The fundamental dilemma in sequential decision-making between gathering new information about unknown regions of the search space (exploration) and leveraging current knowledge to select the best-known configuration (exploitation). An optimal acquisition function implicitly navigates this trade-off. Excessive exploitation leads to premature convergence on a local optimum, while excessive exploration wastes the evaluation budget on unpromising areas.

• Exploration: Sampling points with high predictive variance to reduce model uncertainty.
• Exploitation: Sampling points with a high predicted mean to maximize the objective.
• Regret: The opportunity cost of not choosing the true optimal point at each step.

The broader algorithmic framework that Bayesian optimization instantiates. SMBO iterates between fitting a surrogate model to the history of observations and optimizing an acquisition function to select the next query point. This closed loop allows the algorithm to learn the structure of the objective function and efficiently allocate a limited evaluation budget.

• Step 1: Build a surrogate model from the current observation set.
• Step 2: Use the acquisition function to select the next candidate.
• Step 3: Evaluate the true objective at the candidate.
• Step 4: Update the observation set and repeat.

The most prominent application domain for Bayesian optimization. HPO involves finding the optimal configuration of hyperparameters—such as learning rate, batch size, or network depth—that maximize the validation performance of a machine learning model. Each evaluation requires a full training run, making the objective function expensive to evaluate and non-differentiable. Bayesian optimization consistently outperforms grid search and random search in sample efficiency.

• Use case: Tuning a transformer's learning rate schedule and dropout rate.
• Benefit: Finds high-performing configurations with an order of magnitude fewer trials than random search.