Surrogate Model

The primary purpose of a surrogate model is to be orders of magnitude cheaper to evaluate than the true, expensive objective function (e.g., training a full neural network). This efficiency enables the exploration of vast search spaces that would be otherwise intractable. Common surrogates include:

• Gaussian Processes (GPs): Provide uncertainty estimates.
• Random Forests: Handle high-dimensional, categorical spaces well.
• Bayesian Neural Networks: Offer a deep learning alternative to GPs. By predicting performance, they act as a fast proxy, allowing an optimization algorithm to test thousands of hypothetical configurations in seconds.

A high-quality surrogate model doesn't just predict a single performance score; it outputs a predictive distribution that quantifies uncertainty. This is critical for balancing exploration (searching new, uncertain regions) and exploitation (refining known good areas).

• Mean and Variance: A Gaussian Process outputs a mean prediction and a variance, representing confidence.
• Acquisition Functions: This uncertainty is used by functions like Expected Improvement (EI) or Upper Confidence Bound (UCB) to decide the next most promising configuration to evaluate on the true, expensive function.

Surrogate models are the core of Sequential Model-Based Optimization (SMBO). They operate in a closed-loop, iterative process:

1. Build/Update: Train the surrogate on all observed (configuration, performance) pairs.
2. Propose: Use an acquisition function to select the most promising new configuration.
3. Evaluate: Run the expensive true function (e.g., train model) for that configuration.
4. Augment: Add the new result to the observation set. This loop continues until a budget is exhausted, ensuring each expensive evaluation is strategically chosen based on all prior knowledge.

The objective function being optimized is treated as a black box. The surrogate does not need to understand the internal mechanics of the machine learning model training process. It only learns the input-output mapping:

• Input: A vector representing a hyperparameter configuration or architectural description.
• Output: The resulting validation accuracy, loss, or other metric. This model-agnostic property makes surrogate-based optimization (like Bayesian Optimization) applicable to a wide range of problems beyond AutoML, including scientific simulations and engineering design.

Advanced surrogate models can leverage lower-fidelity approximations to accelerate the search. Instead of only using the expensive, full evaluation, they can learn from cheaper proxies:

• Training for fewer epochs
• Using a subset of data
• Training a smaller model variant The surrogate correlates performance across fidelities, allowing it to cheaply identify promising regions before committing high-fidelity resources. Algorithms like Hyperband and Multi-fidelity Bayesian Optimization build on this principle.

Surrogate models are the engine behind several critical AutoML techniques:

• Hyperparameter Optimization (HPO): Approximating validation loss vs. hyperparameters (e.g., learning rate, batch size).
• Neural Architecture Search (NAS): Modeling the relationship between architectural choices (e.g., number of layers, operation types) and task performance.
• Combined Algorithm Selection and Hyperparameter optimization (CASH): Modeling performance across different algorithm types and their hyperparameters jointly.
• Automated Data Augmentation Policy Search: Finding optimal sequences of image transformations.

Bayesian optimization is the overarching sequential model-based optimization strategy that uses a surrogate model to guide the search for the optimum of an expensive black-box function, such as validation loss. It iteratively:

• Builds a probabilistic surrogate (e.g., a Gaussian Process) from observed evaluations.
• Uses an acquisition function (like Expected Improvement) to select the most promising next hyperparameter configuration to evaluate.
• Updates the surrogate with the new result, refining its approximation of the objective landscape. This creates a balance between exploration (probing uncertain regions) and exploitation (refining known good areas).

An acquisition function is the decision-making engine within Bayesian optimization. After the surrogate model provides a probabilistic prediction (mean and uncertainty), the acquisition function calculates a single score to determine the next hyperparameter set to test. Common functions include:

• Expected Improvement (EI): Measures the expected amount of improvement over the current best observation.
• Upper Confidence Bound (UCB): Favors points with high predicted value or high uncertainty, controlled by a trade-off parameter.
• Probability of Improvement (PI): Simpler than EI, it measures the probability that a point will be better than the current best. The function's maximum dictates the next evaluation point, directly driving the search efficiency.

Multi-fidelity optimization is a strategy to reduce the cost of surrogate model training by using cheaper, lower-fidelity approximations of model performance. Instead of always training a candidate configuration to completion, it uses proxies like:

• Training on a subset of the data.
• Training for a reduced number of epochs.
• Using a smaller, proxy model architecture. The surrogate model learns from these low-fidelity evaluations to predict high-fidelity performance. Algorithms like Hyperband and Successive Halving use this principle to quickly discard poor configurations, allocating full resources only to the most promising ones.

The search space is the formally defined universe of all possible hyperparameter configurations the optimization algorithm can propose. It must be carefully designed to be expressive yet tractable. Key aspects include:

• Parameter Types: Continuous (e.g., learning rate), integer (e.g., number of layers), categorical (e.g., optimizer type).
• Constraints: Defining valid ranges and dependencies between parameters.
• Dimensionality: The number of hyperparameters; high-dimensional spaces are notoriously difficult to search (curse of dimensionality). The surrogate model's job is to approximate the complex, often non-linear relationship between points in this space and the target objective.

A Gaussian Process (GP) is a powerful, non-parametric probabilistic model frequently used as the surrogate in Bayesian optimization. It is preferred because it:

• Provides a full predictive distribution (mean and variance) for any point in the search space, quantifying uncertainty.
• Is defined by a mean function and a kernel (covariance) function. The kernel (e.g., Matern, RBF) encodes assumptions about the smoothness and periodicity of the objective function.
• Is data-efficient, making good predictions with relatively few observations. However, GPs scale cubically with the number of observations, making them less suitable for very high-dimensional spaces or massive evaluation histories, where alternatives like Random Forests or Tree-structured Parzen Estimators (TPE) may be used.

Hyperparameter Optimization is the broader problem that surrogate models are designed to solve. HPO automates the search for the configuration of a model's training process (hyperparameters) that yields the best performance on a validation set. Methods include:

• Grid Search: Exhaustive over a discretized grid; inefficient and non-scalable.
• Random Search: Random sampling; often more efficient than grid search.
• Model-Based Optimization (e.g., Bayesian Opt.): Uses a surrogate model to guide the search, providing superior sample efficiency. Surrogate-based HPO is essential because evaluating a single hyperparameter set often requires training a full model, which is computationally expensive.