Bayesian Optimization

Instead of evaluating the expensive true objective function directly, Bayesian Optimization constructs a cheap-to-evaluate probabilistic surrogate, most commonly a Gaussian Process (GP). This model approximates the objective function and, crucially, provides well-calibrated uncertainty estimates (predictive variance) for every point in the search space. The surrogate is updated with each new observation, refining its belief about the function's shape.

The acquisition function quantifies the utility of evaluating the true objective at any candidate point, balancing the exploration vs. exploitation trade-off. It uses the surrogate's posterior mean and variance to calculate a score. The next point to evaluate is the one that maximizes this function. Common choices include:

• Expected Improvement (EI): Measures the expected gain over the current best observation.
• Upper Confidence Bound (UCB): Optimistically weights the mean by the uncertainty.
• Probability of Improvement (PI): Maximizes the chance of any improvement.

Bayesian Optimization operates in a closed loop:

1. Model: Fit the probabilistic surrogate to all historical observations.
2. Acquire: Use the acquisition function to select the most promising next point to sample.
3. Evaluate: Run the expensive true objective function at the selected point.
4. Update: Add the new observation to the dataset and repeat. This sequential nature makes it highly sample-efficient, ideal for optimizing hyperparameters or physical experiments where each trial is costly.

The most famous application of Bayesian Optimization is in automated machine learning (AutoML). It efficiently searches the space of model hyperparameters (e.g., learning rate, number of layers, regularization strength) to find the configuration that minimizes validation error. Unlike grid or random search, it intelligently focuses on promising regions, often finding superior models in 10-100x fewer trials, saving significant compute cost and time.

In manufacturing and physical sciences, Bayesian Optimization drives self-optimizing experiments. For a chemical vapor deposition process, the algorithm can sequentially suggest the next set of temperature and pressure parameters to test to maximize film quality. It naturally handles noisy observations and can incorporate safety constraints to avoid exploring parameter regions known to damage equipment or produce scrap.

The behavior of the Gaussian Process surrogate is governed by its covariance kernel, which encodes prior assumptions about the function's smoothness and structure. The Matérn kernel is a common choice for modeling realistic physical processes. Selecting an appropriate kernel is critical; a periodic kernel can model seasonal effects, while an additive kernel can capture decoupled influences of different parameters.

The probabilistic surrogate model that underpins most Bayesian optimization. It provides a posterior distribution over functions, offering both a predicted mean and a well-calibrated uncertainty estimate (variance) for any point in the search space. This uncertainty is critical for guiding the acquisition function to balance exploration and exploitation. Unlike neural networks, GPs naturally quantify 'knowing what you don't know,' making them ideal for optimizing expensive black-box functions with sparse data.

The mathematical heuristic that decides where to sample next. It uses the surrogate model's predictions to score the utility of evaluating any candidate point. Common functions include:

• Expected Improvement (EI): Measures the expected gain over the current best observation.
• Upper Confidence Bound (UCB): Optimistically combines the predicted mean and variance.
• Probability of Improvement (PI): Maximizes the chance of finding a better point. The choice of acquisition function directly controls the exploration-exploitation trade-off.

A form of adaptive process control where recipe parameters are modified between processing runs based on post-process metrology. It compensates for slow drift and maintains target quality. Bayesian optimization is a natural fit for R2R on processes with long cycle times and high measurement costs, such as semiconductor wafer fabrication, where it can efficiently learn the optimal recipe adjustments over a minimal number of runs.

A classical statistical methodology for planning efficient experiments to understand factor effects. Traditional DoE uses fixed designs like factorial or response surface methods. Bayesian optimization represents a sequential, model-based alternative that is often more sample-efficient for high-dimensional or non-linear problems. It treats experimental design as an iterative decision-making problem under uncertainty, adapting the design based on all previous outcomes.