Neural Predictor

A neural predictor acts as a surrogate model or performance predictor, approximating the expensive objective function of training a full neural network. It is trained on a dataset of (architecture encoding, performance metric) pairs. Once trained, it can infer the validation accuracy, latency, or other metrics of a new architecture in milliseconds, bypassing hours of GPU training. This is the core mechanism that accelerates the NAS search loop.

The predictor's input is a numerical encoding or representation of the neural network's architecture. Common encodings include:

• Adjacency matrices and operation lists for cell-based NAS.
• Graph neural network (GNN) embeddings that capture connectivity and operation types.
• Path-based feature vectors that enumerate all possible computational paths.
• Learned embeddings from a one-shot supernet. The choice of encoding directly impacts the predictor's ability to generalize to unseen architectures.

Predictors are typically framed as regression models (e.g., predicting exact accuracy) or ranking models (e.g., predicting pairwise architecture preferences).

• Regression uses mean squared error (MSE) loss to predict absolute performance values.
• Ranking uses pairwise hinge or listwise loss to predict relative order, which is often sufficient for search. The predictor's output is a scalar score used to prune poor candidates and select promising ones for further evaluation.

A critical challenge is achieving sample efficiency—making accurate predictions after seeing only a few hundred trained architectures. Predictors must generalize to the vast, discrete search space. Techniques to improve this include:

• Leveraging weight-sharing supernet features as informative encodings.
• Using meta-learning to pre-train on data from related tasks.
• Applying data augmentation in the architecture space (e.g., morphing graphs). Poor generalization leads to search bias and suboptimal final architectures.

The predictor is not used in isolation; it is a component within a larger NAS search strategy. Common integrations include:

• Predictor-based search: The predictor scores a large pool of candidates; the top-K are fully trained.
• Bayesian Optimization: The predictor acts as the surrogate model (e.g., a Gaussian process or neural network) guiding acquisition function decisions.
• Evolutionary algorithms: The predictor provides fast fitness evaluations for population members. Its predictions guide the exploration-exploitation trade-off.

To reduce the cost of creating training data, predictors often use multi-fidelity estimations. Instead of full training, architectures are trained for a few epochs (low-fidelity), and the predictor learns to correlate this with final performance. The most extreme form is a zero-cost proxy, which uses a single forward/backward pass to compute heuristic scores (e.g., gradient norm, synaptic saliency). Neural predictors can be trained to refine or ensemble these cheap proxies for better accuracy.

The core application of a neural predictor is to serve as a performance surrogate within a NAS pipeline. Instead of training thousands of candidate architectures to completion—a process that can take thousands of GPU-days—the predictor provides a rapid, low-cost estimate of final accuracy or another target metric (e.g., latency, model size). This enables the exploration of vast search spaces with orders-of-magnitude less computation.

• Workflow: Architectures are encoded into a fixed-length vector. The predictor, a regression model (e.g., MLP, GNN, or LSTM), takes this encoding and outputs a predicted score.
• Impact: Makes large-scale NAS feasible for organizations without exascale compute resources, democratizing architecture discovery.

Neural predictors are crucial for multi-objective NAS, where the goal is to find architectures that balance competing constraints like accuracy, inference latency, and memory footprint. A predictor can be trained to estimate each objective separately (e.g., one predictor for accuracy, another for latency on target hardware).

• Pareto Frontier Discovery: By querying these predictors, search algorithms can efficiently identify architectures that lie on the Pareto-optimal frontier, representing the best possible trade-offs.
• Hardware-Aware Design: Predictors trained on hardware performance proxies (e.g., measured latency on a specific device) allow for the direct search of architectures optimized for deployment on edge devices, mobile phones, or specialized accelerators.

In one-shot NAS methods, a single, over-parameterized supernet is trained. Evaluating a sub-architecture involves inheriting weights from this supernet. A neural predictor refines this process by learning the correlation between a sub-architecture's encoding and its performance when using the shared supernet weights.

• Ranking Candidates: The predictor learns to rank sub-architectures accurately, identifying which are truly promising for final stand-alone training, even if their one-shot accuracy scores are noisy.
• Reducing Bias: Helps mitigate the weight-sharing bias inherent in one-shot methods, where the performance of sub-architectures within the supernet may not perfectly correlate with their performance when trained independently.

Neural predictors function as highly flexible surrogate models within a Bayesian optimization loop for NAS. Unlike standard Gaussian process surrogates, a neural network predictor can better handle the high-dimensional, structured nature of architecture encodings.

• Acquisition Function: The predictor's uncertainty estimates (if calibrated) can be used by an acquisition function (e.g., Expected Improvement) to balance exploration and exploitation.
• Sequential Design: The loop iterates: predict performance for a batch of candidates, select the most promising via the acquisition function, train them fully to get ground-truth labels, and update the predictor. This creates a highly sample-efficient search.

A well-generalized neural predictor can transfer knowledge across different datasets or tasks. Once trained on a corpus of (architecture, performance) pairs from one domain (e.g., ImageNet classification), it can provide reasonable estimates for architectures applied to a related but novel domain (e.g., a different image dataset), enabling cross-task efficiency.

• Warm-Starting Searches: This capability allows NAS to be warm-started for a new problem, drastically reducing the initial random exploration phase.
• Meta-Learning for NAS: The predictor itself can be meta-learned on performance data from many tasks, allowing it to make few-shot predictions for entirely new tasks, aligning with meta-learning principles.

Beyond pure NAS, neural predictors are used as a fast filtering stage in broader AutoML pipelines. They can screen not only neural architectures but also other pipeline components.

• Hyperparameter Optimization: Predict the final validation score of a model given its hyperparameter configuration and architectural template.
• Early Stopping Proxy: Provide a reliable early forecast of a model's final performance after only a few training epochs, informing decisions to terminate underperforming training runs early, saving substantial compute.
• Zero-Cost Proxies: Some ultra-efficient predictors are based on zero-cost proxies—metrics computable from a single forward/backward pass—and use a small neural network to learn how to best combine these proxies into an accurate performance estimate.

Neural architecture search (NAS) is the automated process of discovering high-performing neural network architectures for a given dataset and task. It treats the network design—choices of layers, connections, and operations—as a search problem.

• Core Goal: Automate architecture engineering, optimizing for metrics like accuracy, latency, or model size.
• Key Challenge: Evaluating a candidate architecture typically requires full training, which is computationally prohibitive.
• Predictor's Role: A neural predictor is a surrogate model used within NAS to estimate an architecture's performance without full training, dramatically accelerating the search.

A surrogate model is a computationally cheap approximation of a complex, expensive-to-evaluate function. In NAS and hyperparameter optimization, it predicts the performance (e.g., validation accuracy) of a model configuration.

• Function: Learns the mapping from an architecture encoding or hyperparameter set to a performance score.
• Types: Can be a Gaussian Process (common in Bayesian Optimization), a random forest, or a neural network (a neural predictor).
• Usage: Guides the search algorithm by suggesting which configurations are most promising to evaluate next, reducing the total number of costly training runs.

The search space in NAS defines the universe of all possible neural network architectures the algorithm can explore. It is a constrained design space of architectural decisions.

• Components: Includes choices like number of layers, types of operations (convolution, attention), filter sizes, and connectivity patterns.
• Encoding: Architectures within this space must be converted into a fixed-length vector or graph representation that a neural predictor can process.
• Impact: The size and design of the search space directly influence the difficulty of the prediction task and the ultimate quality of discovered architectures.

A zero-cost proxy is an ultra-efficient heuristic for estimating neural network quality without any training. It uses a single forward/backward pass or analytical measures.

• Examples: Metrics like gradient norm, synaptic saliency, or NASWOT (a state-free score).
• Comparison to Neural Predictor: While a neural predictor is a trained model, a zero-cost proxy requires no training data. Proxies are faster but often less accurate; predictors are more accurate but require a dataset of (architecture, performance) pairs for training.
• Use Case: Often used for initial candidate screening before applying a more accurate neural predictor.

One-shot NAS is an efficient paradigm where a single, over-parameterized supernet encompassing all candidate operations is trained once. Candidate architectures are evaluated as sub-networks of this supernet using weight sharing.

• Mechanism: The supernet's weights are shared across all sub-architectures. Performance is estimated by activating only the chosen path.
• Neural Predictor Synergy: The accuracy estimates from weight-sharing can be noisy. A neural predictor can be trained on these estimates to provide a more reliable ranking, or used to pre-screen architectures before inheriting weights from the supernet.

Performance estimation strategy refers to the method used to assess the quality of a candidate architecture during NAS. The choice of strategy is a fundamental trade-off between cost and fidelity.

• Full Training: Most accurate, but prohibitively expensive.
• Low-Fidelity Estimates: Using fewer training epochs or smaller datasets. Faster but introduces noise.
• Weight-Sharing: As used in one-shot NAS. Very fast but can suffer from bias.
• Learned Predictor (Neural Predictor): Aims for high fidelity at low cost after an initial training phase. This strategy directly addresses the core computational bottleneck of NAS.