Bayesian Optimization

Instead of directly evaluating the expensive objective function, Bayesian optimization builds a cheap-to-evaluate probabilistic proxy—typically a Gaussian Process (GP). This surrogate models the posterior distribution over the objective function, providing both a predicted mean and a quantified uncertainty (variance) for every point in the search space. The GP is updated with each new observation, refining its belief about where the global optimum lies. Alternative surrogates include Tree-structured Parzen Estimators (TPE) and Random Forest regressors, which handle discrete and conditional hyperparameter spaces more naturally than GPs.

The acquisition function is the decision rule that determines where to sample next by leveraging the surrogate model's predictions. It mathematically formalizes the exploration-exploitation trade-off:

• Expected Improvement (EI): Measures the expected gain over the current best observation, naturally balancing high mean and high uncertainty regions.
• Probability of Improvement (PI): Selects the point with the highest probability of exceeding the current best value.
• Upper Confidence Bound (UCB): Optimistically selects points based on a weighted sum of the predicted mean and standard deviation, controlled by a tunable parameter κ. The acquisition function is maximized—often using gradient-based methods or evolutionary algorithms—to propose the next candidate for evaluation.

Bayesian optimization operates in a closed-loop iterative cycle:

• Step 1: Fit the surrogate model to all previously observed (hyperparameter, objective) pairs.
• Step 2: Use the acquisition function to select the most promising hyperparameter configuration.
• Step 3: Evaluate the true objective function at the proposed configuration—this is the expensive step, such as training a deep neural network.
• Step 4: Append the new observation to the history and repeat. This sequential nature distinguishes it from grid or random search, which are non-adaptive and waste evaluations in unpromising regions. Each iteration actively learns from past results.

The defining advantage of Bayesian optimization is its extreme sample efficiency. It typically finds near-optimal hyperparameters in 10-100x fewer evaluations than grid or random search. This is critical when each function evaluation is expensive—for example, training a large transformer model that takes days and costs thousands of dollars in compute. By reasoning probabilistically about where the optimum might be, Bayesian optimization avoids wasting trials on regions already known to be poor. This makes it the default choice for hyperparameter tuning of deep learning models, simulation-based optimization, and experimental design in wet-lab science.

The behavior of a Gaussian Process surrogate is governed by its covariance kernel, which encodes prior assumptions about the smoothness and structure of the objective function:

• Matérn 5/2 kernel: The most common choice for hyperparameter optimization; assumes the function is twice-differentiable and works well for realistic, non-smooth response surfaces.
• Radial Basis Function (RBF) kernel: Assumes infinite differentiability, which is often too smooth for practical hyperparameter landscapes.
• Dot product kernel: Suitable for modeling linear trends. Kernel hyperparameters—length scales and signal variance—are learned from data by maximizing the marginal likelihood, allowing the surrogate to adapt its smoothness assumptions automatically.

Modern Bayesian optimization frameworks extend beyond unconstrained single-objective optimization:

• Constrained BO: Incorporates feasibility constraints by modeling constraint functions with separate GPs and weighting the acquisition function by the probability of feasibility.
• Multi-objective BO: Uses Expected Hypervolume Improvement (EHVI) to find the Pareto frontier when optimizing multiple competing objectives simultaneously.
• Multi-fidelity BO: Leverages cheap, low-fidelity approximations—such as training with fewer epochs or on a subset of data—to accelerate the search before committing to expensive full evaluations. Frameworks like Hyperband combine multi-fidelity early stopping with Bayesian model-based selection.

A probabilistic approximation of the true objective function, typically a Gaussian Process (GP) , that is computationally cheap to evaluate. The surrogate models the mean and uncertainty of the objective at every point in the search space.

• Gaussian Process: The most common surrogate, defined by a mean function and a covariance kernel (e.g., Matérn 5/2)
• Tree-structured Parzen Estimator (TPE) : An alternative surrogate that models P(x|y) and P(y) rather than P(y|x), used in Hyperopt
• Updated sequentially with each new observation, refining its posterior distribution

A utility function that guides the search by quantifying the value of sampling at any candidate point, balancing exploration (high uncertainty) against exploitation (high predicted performance).

• Expected Improvement (EI) : Measures the expected gain over the current best observation, the most widely used function
• Probability of Improvement (PI) : Maximizes the probability of beating the incumbent best
• Upper Confidence Bound (UCB) : Explicitly trades off mean and variance with a tunable parameter κ
• Entropy Search: Selects points that maximally reduce uncertainty about the location of the global optimum

The covariance function that encodes prior assumptions about the smoothness, periodicity, and structure of the objective function. The kernel choice critically determines the surrogate model's inductive bias.

• Matérn 5/2: Assumes twice-differentiable functions, a robust default for realistic objective landscapes
• Radial Basis Function (RBF) : Assumes infinitely differentiable, very smooth functions
• Periodic Kernel: Captures cyclical patterns, useful for seasonality in financial parameters
• Automatic Relevance Determination (ARD) kernels learn separate length scales per input dimension

The iterative framework that underpins Bayesian optimization. Each iteration fits the surrogate model to all observed data, maximizes the acquisition function to select the next point, evaluates the true objective, and updates the dataset.

• Initialization: Start with a small set of random evaluations via Latin Hypercube or Sobol sampling
• Loop: Fit surrogate → Optimize acquisition → Evaluate objective → Append observation
• Termination: Stop after a fixed budget, when improvement plateaus, or when a target threshold is reached
• In trading, each evaluation might be a full backtest run, making sample efficiency paramount

The primary application of Bayesian optimization in machine learning: automatically finding the optimal configuration of model parameters that are not learned from data, such as learning rates, regularization strengths, and architecture choices.

• Tuning a deep LSTM for tick data: Optimize hidden units, dropout rate, sequence length, and learning rate simultaneously
• XGBoost for alpha factors: Tune max_depth, subsample, colsample_bytree, and min_child_weight
• Handles mixed variable types: continuous (learning rate), integer (number of layers), and categorical (activation function)
• Outperforms random search and grid search when each evaluation is expensive, as in backtesting

An extension that leverages cheap, low-fidelity approximations of the objective to accelerate the search before committing to expensive, high-fidelity evaluations. Critical when a full backtest is computationally prohibitive.

• Successive Halving: Dynamically allocates resources, terminating poorly performing configurations early
• Hyperband: Combines random search with successive halving, providing strong anytime performance
• BOHB (Bayesian Optimization + Hyperband) : Replaces the random sampling in Hyperband with a TPE-guided Bayesian search
• In finance, low-fidelity could be a short backtest period or a subset of assets; high-fidelity is the full simulation