Hyperparameter Optimization (HPO)

A sequential model-based optimization strategy for finding the global optimum of an expensive black-box function, like validation loss. It builds a probabilistic surrogate model (typically a Gaussian Process) to approximate the objective. An acquisition function, such as Expected Improvement, uses this model to balance exploration of uncertain regions and exploitation of known good areas, deciding the next hyperparameter configuration to evaluate. It is significantly more sample-efficient than random or grid search for costly evaluations.

• Key Component: Surrogate Model & Acquisition Function
• Best For: Expensive-to-evaluate functions with moderate-dimensional search spaces.
• Example Tools: Scikit-optimize, Ax, BoTorch.

A strategy that uses cheaper, lower-fidelity approximations of model performance to guide the search for configurations that perform well at high fidelity. This dramatically reduces total optimization cost. Common low-fidelity proxies include:

• Training on a subset of the data.
• Training for a reduced number of epochs.
• Using a lower-resolution model.

Algorithms like Hyperband and Successive Halving are foundational multi-fidelity methods. They dynamically allocate training resources (budget) to many configurations in parallel, quickly discarding poor performers and promoting promising ones to higher budgets.

A hybrid algorithm that simultaneously trains and optimizes a population of models. It combines ideas from evolutionary algorithms and traditional training. During parallel training, PBT periodically:

1. Exploits by having poorly performing models copy the weights from better performers.
2. Explores by randomly perturbing (mutating) the hyperparameters of the copied model.

This allows hyperparameters to be optimized online, adapting them as training progresses. It is particularly effective for optimizing schedules (e.g., learning rate decay) and is a core method in large-scale reinforcement learning.

Treats hyperparameters as differentiable parameters and computes their gradients with respect to a validation loss, enabling direct optimization via gradient descent. This requires solving a bilevel optimization problem: optimizing hyperparameters in the outer loop while training model weights in the inner loop. Techniques include:

• Implicit Differentiation: Computes gradients by leveraging the optimality conditions of the inner loop.
• Forward-Mode Differentiation: Unrolls the training loop.

It is highly efficient for hyperparameters that directly influence the gradient-based training process, such as learning rates and regularization strengths.

A family of optimization algorithms inspired by biological evolution. A population of hyperparameter configurations is iteratively evolved through:

• Selection: Choosing the best-performing configurations as parents.
• Crossover (Recombination): Combining parameters from two parents to create offspring.
• Mutation: Randomly altering parameters in offspring to maintain diversity.

This population-based approach is highly parallelizable, robust to noisy evaluations, and can effectively explore complex, non-differentiable, and mixed search spaces (continuous, integer, categorical). It is a foundational technique in Evolutionary NAS.

The Combined Algorithm Selection and Hyperparameter optimization (CASH) problem is the core challenge in AutoML: jointly selecting the best ML algorithm and its optimal hyperparameters from a portfolio. Solving CASH requires defining a hierarchical search space that encompasses different algorithms (e.g., SVM, Random Forest, XGBoost) and their respective hyperparameters.

• Search Space Definition: The set of all possible configurations an HPO algorithm can propose. A well-designed space is critical for efficiency.
• Warm-Starting: Initializing the search with known good configurations from prior tasks to accelerate convergence.
• Tool Example: Auto-sklearn, which uses Bayesian optimization over a CASH space with meta-learning for warm-starting.

Automated Machine Learning (AutoML) is the end-to-end process of automating the application of machine learning to real-world problems. It encompasses and orchestrates multiple stages beyond HPO, including:

• Data preprocessing and feature engineering
• Algorithm selection from a portfolio of models
• Hyperparameter optimization for the selected algorithm
• Model evaluation and pipeline construction

The core problem is often formalized as CASH (Combined Algorithm Selection and Hyperparameter optimization), where the system must jointly find the best learning algorithm and its optimal settings.

Neural Architecture Search (NAS) is a specialized subfield of AutoML focused on automating the design of neural network architectures. While HPO tunes parameters like learning rate, NAS searches over architectural components:

• Cell types (e.g., convolutional, attention)
• Layer connectivity and depth
• Channel widths and expansion ratios

Key methodologies include:

• Reinforcement Learning-based NAS: An RNN controller proposes architectures rewarded by their trained performance.
• Evolutionary NAS: Uses genetic algorithms for population-based architecture evolution.
• Differentiable NAS (DARTS): Relaxes the search space to be continuous, enabling gradient-based optimization.
• One-Shot NAS & Weight Sharing: Evaluates candidates via a shared supernet to reduce computational cost.

Bayesian Optimization (BO) is a leading sequential model-based strategy for global optimization of expensive black-box functions, making it highly effective for HPO. Its core loop is:

1. Build a probabilistic surrogate model (typically a Gaussian Process) to approximate the objective function (e.g., validation loss).
2. Use an acquisition function (e.g., Expected Improvement, Upper Confidence Bound) to select the next hyperparameter configuration to evaluate by balancing exploration (uncertain regions) and exploitation (known promising areas).
3. Evaluate the chosen configuration, update the surrogate model, and repeat.

BO is sample-efficient, making it ideal when evaluating a full model training run is computationally costly.

Meta-Learning, or "learning to learn," is a paradigm where a model is trained on a distribution of tasks so it can rapidly adapt to new tasks with minimal data. It provides a powerful prior for HPO and NAS. Key approaches include:

• Model-Agnostic Meta-Learning (MAML): Learns a model initialization that is sensitive to gradient updates, enabling fast fine-tuning.
• Meta-Learning for Hyperparameter Initialization (Warm-Starting): Uses knowledge from prior tasks to suggest good hyperparameter starting points, dramatically reducing the search budget for a new task.
• Learning Optimizers: Meta-learns the update rule for neural network weights or hyperparameters themselves.

Meta-learning shifts the focus from optimizing for a single task to preparing a system for efficient adaptation.

Population-Based Training (PBT) is a hybrid, asynchronous optimization algorithm that simultaneously trains and tunes a population of models. It combines ideas from evolutionary algorithms and traditional training:

• A population of models trains in parallel.
• Periodically, poorly-performing models exploit by copying the weights of better performers.
• These models then explore by randomly perturbing (mutating) their hyperparameters.

This allows hyperparameters to be optimized online during training itself, rather than in a separate outer loop. PBT is particularly effective for optimizing non-stationary parameters like learning rate schedules and data augmentation policies during a single training run.