Bayesian Optimization

Instead of directly evaluating the expensive objective function, Bayesian optimization builds a probabilistic surrogate model—typically a Gaussian Process (GP)—that approximates the function's behavior. This model provides both a mean prediction and a quantified uncertainty (variance) for every point in the search space. The GP is updated with each new observation, refining its belief about the function's landscape. Alternative surrogates include Tree-structured Parzen Estimators (TPE) and Random Forests, which handle discrete and conditional hyperparameters more naturally than GPs.

The acquisition function guides the search by quantifying the utility of evaluating any candidate point. It balances the exploration-exploitation trade-off:

• Expected Improvement (EI): Measures the expected gain over the current best observation, naturally balancing exploration and exploitation.
• Upper Confidence Bound (UCB): Explicitly weights the mean prediction against the uncertainty with a tunable parameter.
• Probability of Improvement (PI): Selects points most likely to improve upon the current optimum, though it can be overly exploitative. The acquisition function is cheap to evaluate and is itself optimized to propose the next sampling location.

Bayesian optimization operates in a closed-loop iterative cycle:

1. Fit the surrogate model to all observed data points.
2. Optimize the acquisition function to find the most promising candidate.
3. Evaluate the true objective function at that candidate.
4. Update the dataset with the new observation and repeat. This sequential nature distinguishes it from grid or random search, which do not learn from prior evaluations. Each step deliberately reduces uncertainty in regions likely to contain the global optimum.

The defining advantage of Bayesian optimization is its extreme sample efficiency. It typically finds near-optimal solutions in tens to hundreds of evaluations, compared to thousands or millions required by evolutionary algorithms or random search. This makes it indispensable when each function evaluation is costly:

• Hyperparameter tuning of deep neural networks where each trial requires hours of GPU training.
• Simulation-based optimization where a single digital twin run consumes significant compute.
• Physical experiments in materials science or drug discovery with limited lab capacity.

The kernel (covariance function) of the Gaussian Process encodes prior assumptions about the objective function's smoothness and structure. Critical choices include:

• Radial Basis Function (RBF): Assumes infinitely differentiable, smooth functions.
• Matérn kernel: Models less smooth functions, with a parameter controlling differentiability.
• Periodic kernel: Captures cyclical patterns in the objective. Kernel selection and hyperparameter estimation (via marginal likelihood maximization) are essential for accurate uncertainty quantification and effective exploration.

Real-world optimization often involves unknown feasibility constraints that are also expensive to evaluate. Constrained Bayesian optimization extends the framework by:

• Modeling each constraint with its own independent surrogate model.
• Modifying the acquisition function to penalize or exclude points likely to violate constraints.
• Computing the Expected Feasible Improvement (EFI), which multiplies the expected improvement by the probability of satisfying all constraints. This is critical for supply chain simulations where certain parameter combinations represent physically impossible or operationally invalid configurations.

A probabilistic approximation of the expensive objective function, most commonly a Gaussian Process (GP). The surrogate model provides both a predicted mean and a quantified uncertainty (variance) for any point in the search space. Because evaluating the true function is costly, the surrogate acts as a cheap stand-in, guiding the search by suggesting where the true optimum might lie and where uncertainty remains high.

A mathematical function that uses the surrogate model's predictions to score the utility of evaluating any candidate point. It balances exploitation (sampling where the predicted mean is high) against exploration (sampling where the predicted uncertainty is high). Common functions include:

• Expected Improvement (EI): Measures the expected gain over the current best observation.
• Upper Confidence Bound (UCB): Optimistically weights the mean by the uncertainty.
• Probability of Improvement (PI): Maximizes the chance of any improvement.

A non-parametric, probabilistic model defined by a mean function and a covariance kernel. The kernel encodes assumptions about the function's smoothness and periodicity. A GP defines a distribution over functions, and conditioning it on observed data yields posterior predictions with well-calibrated uncertainty estimates. This analytical tractability makes GPs the default surrogate for Bayesian optimization, though alternatives like Random Forests and Tree-structured Parzen Estimators (TPE) are also used.

The fundamental dilemma in sequential optimization. Exploitation focuses on refining the search near the current best-known point, risking convergence to a local optimum. Exploration samples in high-uncertainty regions to discover potentially better global optima, but may waste evaluations on unpromising areas. The acquisition function's parameterization (e.g., the kappa in UCB or xi in EI) explicitly controls this balance, making Bayesian optimization inherently no-regret over many iterations.

The most widespread application of Bayesian optimization. It treats a machine learning model's validation performance as a black-box function of its hyperparameters (learning rate, layer count, regularization strength). Each trial involves a full training run, which is computationally expensive. Bayesian optimization efficiently finds high-performing configurations in far fewer trials than grid search or random search, often achieving comparable results with 10-100x fewer evaluations.