Bayesian Optimization

Instead of directly evaluating the expensive objective function, Bayesian Optimization constructs a probabilistic surrogate model—typically a Gaussian Process (GP). This model approximates the unknown function and, crucially, quantifies the epistemic uncertainty of its predictions. The surrogate provides both a mean estimate and a confidence interval for any point in the search space, enabling informed decisions about where to sample next without exhaustive grid search.

The acquisition function guides the search by assigning a numerical utility to each candidate point, balancing the exploration vs. exploitation trade-off:

• Expected Improvement (EI): Measures the expected gain over the current best observation.
• Upper Confidence Bound (UCB): Optimistically weights points with high uncertainty.
• Probability of Improvement (PI): Selects points most likely to beat the incumbent optimum. The optimizer selects the point that maximizes this function, which is cheap to evaluate compared to the true objective.

The optimization loop operates iteratively:

1. Fit the surrogate model to all historical observations.
2. Maximize the acquisition function to propose the next sampling location.
3. Evaluate the true expensive objective function at that point.
4. Update the dataset with the new observation and repeat. This sequential nature makes it highly sample-efficient, often converging to a global optimum in tens of evaluations rather than the thousands required by random search or grid search.

In smart grid applications, Bayesian Optimization is used to tune complex forecasting models such as Gradient Boosting Machines or LSTMs that predict renewable generation and load. Key hyperparameters optimized include:

• Learning rate and tree depth for XGBoost models.
• Number of hidden layers and dropout rates for neural networks.
• Window size for time-series feature engineering. This avoids the prohibitive cost of cross-validating every combination, directly reducing model training time while improving forecast accuracy for day-ahead market bidding.

The behavior of the Gaussian Process surrogate is governed by its covariance kernel, which encodes prior assumptions about the function's smoothness and periodicity:

• Radial Basis Function (RBF): Assumes infinitely differentiable, smooth functions.
• Matérn Kernel: Provides flexibility over differentiability, often preferred for realistic physical processes.
• Periodic Kernel: Captures cyclical patterns like daily or seasonal load curves. Selecting an appropriate kernel is critical for accurate uncertainty calibration in grid optimization contexts.

Grid optimization problems often involve hard physical constraints, such as thermal limits on transformers or voltage bounds on feeders. Constrained Bayesian Optimization extends the framework by modeling both the objective and constraint functions with separate Gaussian Processes. The acquisition function is then modified to only consider feasible regions where the probability of constraint violation is below a specified threshold, ensuring safe exploration of the parameter space without proposing configurations that would cause a line overload.

A computationally cheap probabilistic approximation of the expensive black-box objective function. In Bayesian optimization, a Gaussian Process (GP) is the standard surrogate, providing both a mean prediction and a quantified uncertainty (variance) at every point in the search space. The surrogate is refit after every trial, guiding the search toward promising regions without evaluating the true function.

A heuristic function that uses the surrogate model's posterior distribution to score candidate evaluation points, balancing the exploration vs. exploitation trade-off. Common functions include:

• Expected Improvement (EI): Measures the expected gain over the current best observation.
• Upper Confidence Bound (UCB): Optimistically weights points with high uncertainty.
• Probability of Improvement (PI): Maximizes the chance of any improvement.

A non-parametric Bayesian model defined by a mean function and a covariance kernel (e.g., Matérn or RBF). The kernel encodes smoothness assumptions about the objective function. A GP defines a distribution over functions, allowing it to output a full predictive distribution (mean and variance) for any unseen input, making it the ideal surrogate for sequential decision-making under uncertainty.

The fundamental dilemma in sequential optimization. Exploration samples regions with high surrogate uncertainty to improve the global model accuracy. Exploitation samples regions with a high predicted mean to refine a known good solution. The acquisition function's hyperparameters (e.g., the kappa or xi term) directly control this trade-off to prevent premature convergence to a local optimum.

The primary application context for Bayesian optimization in machine learning. It automates the search for optimal model configurations (e.g., learning rate, batch size, number of layers) that minimize validation loss. Unlike Grid Search or Random Search, Bayesian optimization uses past trial results to intelligently suggest the next hyperparameter set, often finding superior configurations in significantly fewer iterations.

The formal algorithmic framework underlying Bayesian optimization. The loop iterates through four steps:

1. Build a surrogate model of the objective using all observed data.
2. Optimize the acquisition function to find the next candidate point.
3. Evaluate the true expensive objective at that point.
4. Update the observation set and repeat. This contrasts with parallel or one-shot optimization methods.