Multi-Fidelity Optimization

Trains the model on a randomly selected subset of the full training dataset (e.g., 10%, 50%). This is the most common low-fidelity proxy.

• Mechanism: Reduces computational cost linearly with the sample size.
• Use Case: Initial screening in Hyperband to quickly discard poor hyperparameter configurations.
• Risk: Performance ranking on a small subset may not correlate perfectly with full-data performance, especially for small datasets.

Evaluates model performance after training for a reduced number of epochs instead of running to full convergence.

• Mechanism: Capitalizes on the observation that relative performance between configurations often stabilizes early.
• Example: A configuration trained for 10 epochs often maintains its rank relative to others trained for 100 epochs.
• Application: Core to successive halving algorithms, where budgets are defined in terms of epochs or iterations.

Uses downsampled or lower-fidelity versions of input data, particularly for vision and signal processing tasks.

• Mechanism: Reduces the spatial dimensions of images (e.g., 224x224 → 32x32) or the sampling rate of audio, drastically cutting compute per forward/backward pass.
• Example: In Hardware-Aware NAS for mobile vision, architectures can be initially evaluated on low-res images to estimate efficiency.
• Consideration: May not capture performance on fine-grained details critical for the final task.

Evaluates neural architectures by inheriting weights from a single, shared supernet without standalone training. A cornerstone of efficient Neural Architecture Search (NAS).

• Mechanism: In One-Shot NAS or DARTS, candidate architectures are subgraphs of a supernet. Performance is estimated via a single minibatch forward pass or by training the supernet once.
• Benefit: Reduces architecture evaluation cost from GPU-days to seconds or minutes.
• Limitation: The ranking quality depends heavily on the supernet training protocol and may suffer from optimization bias.

Estimates architecture potential using analytical metrics derived from the network at initialization, requiring no training.

• Common Proxies: Metrics like gradient norm, synaptic flow (SNIP), or Fisher information.
• Process: A single forward/backward pass with a batch of data calculates a score correlating with trained accuracy.
• Application: Used for pre-filtering thousands of architectures in large search spaces before any resource-intensive training begins.

Uses a cheap-to-evaluate machine learning model (the surrogate) to predict the performance of untested hyperparameter configurations.

• Mechanism: A model like a Gaussian Process or Random Forest is trained on {configuration, low-fidelity score} pairs. It predicts scores for new configurations, guiding the search in Bayesian Optimization.
• Multi-Fidelity Extension: Methods like BOHB combine Hyperband with a surrogate model that learns across all fidelities.
• Advantage: Enables intelligent, sample-efficient search by modeling the performance landscape.

In aerospace engineering, optimizing wing or turbine blade shapes involves evaluating thousands of designs. Multi-fidelity optimization is critical here:

• Low-fidelity: Fast computational fluid dynamics (CFD) simulations with coarse meshes or simplified physics.
• Medium-fidelity: Simulations with refined meshes.
• High-fidelity: High-resolution CFD or physical wind tunnel tests. The algorithm uses cheap low-fidelity evaluations to discard poor designs, only investing in high-fidelity simulations for the most promising candidates, reducing total compute time from months to weeks.

Identifying promising drug candidates requires evaluating millions of molecular structures for properties like binding affinity. Multi-fidelity proxies accelerate this search:

• Low-fidelity: Quick, approximate scoring functions or machine learning models trained on large, lower-quality data.
• Medium-fidelity: More accurate molecular docking simulations.
• High-fidelity: Experimental wet-lab assays or free-energy perturbation calculations. This approach allows researchers to explore a vast chemical space in-silico before committing to expensive lab experiments.

Designing efficient neural networks for mobile phones or embedded devices requires balancing accuracy with latency and power. Multi-fidelity optimization enables practical NAS:

• Low-fidelity: Evaluating architecture accuracy on a small proxy dataset (e.g., CIFAR-10) or using zero-cost proxies that require no training.
• Medium-fidelity: Training candidate architectures on the target dataset for a reduced number of epochs.
• High-fidelity: Full training of the top-performing architectures, followed by precise latency measurement on the target hardware (e.g., a smartphone). This process efficiently navigates the massive architecture search space under real-world constraints.

Discovering new materials with specific properties (strength, conductivity) involves costly experiments. Multi-fidelity methods create an efficient discovery pipeline:

• Low-fidelity: High-throughput computational screening using density functional theory (DFT) approximations.
• Medium-fidelity: More accurate, but slower, DFT calculations.
• High-fidelity: Physical synthesis and laboratory characterization of the most promising candidates. The surrogate model built from computational data guides experiments, minimizing the number of real-world tests needed.

The automated process of searching for the optimal set of hyperparameters (e.g., learning rate, batch size, network depth) that govern a model's training to maximize performance. Multi-fidelity optimization is a specific, efficient strategy within HPO.

• Core Objective: Minimize validation error by tuning model settings.
• Common Methods: Grid search, random search, Bayesian optimization.
• Key Challenge: The high cost of evaluating each configuration at full fidelity.

A sequential model-based optimization strategy for finding the optimum of expensive black-box functions, like validation loss. It builds a probabilistic surrogate model (e.g., Gaussian Process) to predict performance and uses an acquisition function to guide the search.

• Surrogate Model: Approximates the true objective function.
• Acquisition Function: Balances exploration and exploitation (e.g., Expected Improvement).
• Relation to Multi-Fidelity: Often used as the core optimizer in multi-fidelity methods, where the surrogate model incorporates fidelity as an input dimension.

A multi-fidelity optimization algorithm that uses successive halving and adaptive resource allocation. It aggressively eliminates poor-performing configurations early using low-fidelity evaluations (e.g., few epochs), then allocates more resources to promising candidates.

• Successive Halving: At each bracket, the worst half of configurations are discarded.
• Adaptive Resource Allocation: Dynamically decides how many configurations to evaluate and at what fidelity.
• Key Advantage: Provides a robust, non-parametric baseline that doesn't require a surrogate model.

The automated search for high-performing neural network architectures. NAS is computationally intensive, making multi-fidelity techniques essential. Methods like weight-sharing (e.g., in One-Shot NAS) are a form of low-fidelity evaluation.

• Search Space: Defines possible operations (convolution, pooling) and connections.
• Search Strategy: Reinforcement learning, evolutionary algorithms, gradient-based methods (DARTS).
• Fidelity in NAS: Low fidelity can mean training on a subset of data, for fewer epochs, or with weight sharing in a supernet.

A computationally cheap model trained to approximate the expensive objective function (e.g., validation accuracy vs. hyperparameters). In multi-fidelity optimization, the surrogate model is often extended to also model the relationship between performance, configuration, and fidelity level.

• Common Types: Gaussian Processes, Random Forests, Neural Networks.
• Multi-Fidelity Extension: Models like Multi-Task Gaussian Processes learn correlations across fidelities.
• Purpose: Enables prediction of high-fidelity performance from cheaper low-fidelity runs, guiding the search efficiently.

The overarching field of automating the end-to-end machine learning pipeline. Multi-fidelity optimization is a critical component for the hyperparameter tuning and model selection stages within an AutoML system.

• Scope: Includes automated data preprocessing, feature engineering, model selection (CASH), and hyperparameter optimization.
• CASH Problem: Combined Algorithm Selection and Hyperparameter optimization.
• System Goal: Reduce the need for human ML expertise and iteration time, making ML more accessible and efficient.