Zero-Cost Proxies

These proxies analyze the gradient flow or magnitude after a single backward pass on a small batch of data. The underlying hypothesis is that architectures with well-behaved gradients at initialization are easier to optimize.

• GradNorm: Measures the norm of the gradients. Architectures with very small or exploding gradients may train poorly.
• GradSign: Analyzes the consistency of gradient directions across examples. Greater consistency can indicate a more stable optimization landscape.
• SNIP (Single-shot Network Pruning): Estimates parameter sensitivity by approximating the effect of removing each weight on the loss, based on a single gradient computation.

These proxies estimate the representational power or complexity of an architecture based on its structure at initialization.

• SynFlow (Synaptic Flow): A pruning-inspired score that measures the total flow of information through the network using a uniform input (all ones). It helps identify architectures with sufficient expressive capacity while avoiding over-parameterization.
• # of Linear Regions: For networks with ReLU activations, this estimates the number of distinct linear functions the network can represent, a theoretical measure of its complexity.
• Effective Capacity: Metrics based on the Fisher Information or the NTK (Neural Tangent Kernel) at initialization, which relate to the model's trainability and convergence behavior.

These proxies use the output of a single forward pass on random or real data to make a prediction about final trained accuracy.

• NASWOT (NAS Without Training): Uses the number of linear regions and the kernel matrix of activations from a forward pass to predict accuracy. It correlates the kernel's rank with network trainability.
• Zen-Score: Computes a score from the Gaussian process of the network's outputs over multiple random inputs, capturing the network's behavior distribution without training.
• Initialization Loss: The loss value after the first forward pass (with random weights). While crude, extremely high initial loss can be a negative signal.

Simpler, often hand-designed, metrics based on counting architectural elements or their relationships. These are the fastest to compute.

• # of Parameters / FLOPs: While not direct performance predictors, they are used as constraints in hardware-aware NAS. The goal is to find the best-performing architecture within a FLOP or parameter budget.
• Depth / Width Ratios: Heuristics about optimal layer scaling, e.g., the ratio of filter counts in consecutive convolutional layers.
• Skip Connection Analysis: Counting or analyzing the pattern of residual connections, which are known to ease gradient flow and improve training.

A critical meta-category: the process of measuring how well a zero-cost proxy correlates with final trained accuracy across many architectures. No proxy is universally perfect.

• Kendall's Tau / Spearman's Rank: Statistical measures used to evaluate the rank correlation between the proxy scores and the true performance ranking of architectures.
• Task/Data Dependency: A proxy that works well on CIFAR-10 may not generalize to ImageNet or NLP tasks. Validation on the target domain's search space is essential.
• Ensemble of Proxies: Combining multiple zero-cost metrics (e.g., GradNorm + SynFlow) often yields a more robust and correlated predictor than any single one.

How zero-cost proxies are integrated into practical Neural Architecture Search (NAS) workflows to achieve massive speed-ups.

• Pre-filtering: Screening thousands of candidate architectures to select a promising subset (e.g., top 10%) for full training.
• Guiding Evolutionary Search: Using the proxy as a fast fitness function to steer mutation and crossover operations in Evolutionary NAS.
• Warm-Starting Bayesian Optimization: Using proxy scores to initialize the surrogate model in Bayesian Optimization, leading to faster convergence.
• Weight-Sharing Supernet Evaluation: In One-Shot NAS, zero-cost proxies can be used to score sub-networks sampled from the supernet, providing a training-free alternative to the standard weight-sharing evaluation.

Zero-cost proxies are integral to one-shot NAS methodologies. A single supernet is trained once, and candidate sub-networks (architectures) are evaluated by inheriting weights from this supernet. Proxies like grad_norm or synflow are computed on these inherited weights in a single forward/backward pass, providing a performance score without any independent training. This replaces the need for resource-intensive individual candidate training, reducing search time from thousands of GPU hours to a matter of hours or less.

• Key Technique: Weight sharing within a supernet.
• Common Proxy: synflow (Synaptic Flow), which identifies well-connected subnetworks.
• Outcome: Orders-of-magnitude faster architecture discovery.

In hardware-aware NAS, the goal is to find architectures that balance accuracy with constraints like latency, memory, or energy use. Zero-cost proxies for hardware metrics are used alongside performance proxies. For example, a proxy for inference latency can be estimated by analyzing the architecture's graph structure (e.g., FLOPs, parameter count, layer types) without deploying it on target hardware. This allows the search algorithm to efficiently navigate the Pareto front of optimal accuracy-latency trade-offs.

• Multi-Objective Search: Optimizes for both accuracy (via a zero-cost proxy) and a hardware metric.
• Efficiency: Avoids costly on-device profiling for every candidate.
• Application: Designing models for mobile phones, embedded systems, and edge devices.

Zero-cost proxies are used to warm-start more expensive search algorithms like Bayesian optimization or evolutionary algorithms. By scoring a large random sample of architectures from the search space with a zero-cost proxy, the search can be initialized with the top-k promising candidates, bypassing the initial random exploration phase. Furthermore, proxies can prune the search space by eliminating architectures with very low proxy scores before any expensive evaluation begins, focusing computational budget on the most promising regions.

• Process: Rank thousands of random architectures using a proxy, then feed the best into a refined search.
• Benefit: Dramatically improves the sample efficiency of subsequent optimization.
• Outcome: Higher final performance for a given computational budget.

Different zero-cost proxies capture distinct theoretical properties of an initialized network. They are broadly categorized by the information they leverage:

• Gradient-Based Proxies: Such as grad_norm and snip (SNIP: Single-shot Network Pruning). These compute the magnitude of gradients after a single batch, hypothesizing that architectures where parameters have larger gradients will train more effectively.
• Pruning-Based Proxies: Like synflow and grasp. These evaluate the importance of connections or neurons, often using criteria from pruning literature, under the assumption that well-connected, trainable networks score highly.
• Expressivity/Information-Theoretic Proxies: Such as jacob_cov (Jacobian covariance), which measures the output variability with respect to input changes, correlating with network expressivity.

No single proxy is universally best; performance depends on the dataset and search space.

Zero-cost proxies represent the lowest-fidelity evaluation in a multi-fidelity optimization hierarchy. In frameworks like Hyperband, evaluations start at the lowest fidelity (zero-cost proxy score). Only the most promising architectures are promoted to higher, more expensive fidelities, such as training for a few epochs (low-fidelity training) or to completion (high-fidelity validation). This creates a cascading filter:

1. Stage 1 (Zero-Cost): Score all candidates with a proxy (e.g., synflow).
2. Stage 2 (Low-Fidelity): Train top candidates for 5-10 epochs.
3. Stage 3 (High-Fidelity): Fully train the final few contenders.

This systematic resource allocation prevents wasting compute on poor architectures.

While powerful, zero-cost proxies have key limitations that dictate their practical use:

• Rank Correlation, Not Absolute Accuracy: Proxies are designed to rank architectures correctly relative to each other, not to predict their exact final accuracy. A high proxy score indicates potential, not a guarantee.
• Search Space Dependence: A proxy that works well on one search space (e.g., for convolutional networks) may not transfer effectively to another (e.g., for transformers).
• Dataset Bias: Proxy scores can be sensitive to the statistics of the small batch of data used for the forward/backward pass. Using multiple batches can stabilize estimates.
• Not a Replacement for Validation: The final selected architecture must still be fully trained and validated. Proxies are a pre-filter, not a final evaluator.

Best practice involves validating the chosen proxy's ranking correlation on a small subset of a new search space before full-scale deployment.

One-Shot NAS is a family of methods that encode the entire search space within a single, over-parameterized supernet. Candidate architectures are evaluated as subgraphs of this supernet.

• Weight Sharing: All candidate models share parameters from the supernet. A candidate is evaluated by performing inference on the subgraph using these shared weights.
• Efficiency Gain: This avoids training each candidate from scratch, reducing search cost from thousands of GPU days to a few.
• Limitation: The accuracy estimate from shared weights is a proxy for the true performance of an independently trained model. Zero-cost proxies can be seen as an extreme form of this, where even the supernet training is eliminated, relying solely on properties at initialization.

Hyperparameter Optimization (HPO) is the automated search for the optimal set of hyperparameters that control a model's training process, such as learning rate, batch size, or optimizer settings.

• Shared Objective with NAS: Both HPO and NAS solve a black-box optimization problem where evaluating a configuration (hyperparameters or architecture) is expensive.
• Key Techniques: Includes Bayesian Optimization, Random Search, and Hyperband.
• Distinction: NAS searches over architectural structure, while HPO searches over training parameters. They are often combined in the CASH (Combined Algorithm Selection and Hyperparameter optimization) problem.
• Proxy Use: Multi-fidelity optimization in HPO uses cheaper, low-fidelity proxies (e.g., training for few epochs). Zero-cost proxies represent the ultimate low-fidelity approximation.

Hardware-Aware NAS incorporates target deployment constraints—like latency, memory footprint, or energy consumption—directly into the architecture search objective.

• Multi-Objective Optimization: The goal is to find architectures on the Pareto front, balancing accuracy against a hardware metric.
• Search Cost Problem: Evaluating hardware metrics (e.g., on-device latency) for each candidate is also computationally expensive.
• Synergy with Proxies: Zero-cost proxies can estimate hardware metrics extremely quickly. For example, synaptic flow or gradient-based metrics can correlate with final parameter count or FLOPs, allowing the search to pre-filter architectures that violate constraints before any costly measurement.

Bayesian Optimization (BO) is a sequential strategy for optimizing expensive black-box functions. It uses a probabilistic surrogate model (like a Gaussian Process) to model the objective function (e.g., validation accuracy).

• Acquisition Function: Guides the search by balancing exploration and exploitation to select the next configuration to evaluate.
• Traditional Role: The surrogate model is built from actual expensive evaluations (trained model performance).
• Zero-Cost Integration: Zero-cost proxy scores can be used to warm-start or pre-screen the Bayesian Optimization process. They provide cheap, noisy observations to build an initial surrogate model, allowing BO to focus its expensive evaluations on the most promising regions of the search space from the outset.

Differentiable Architecture Search (DARTS) formulates the NAS search space as a continuous relaxation, where the choice between operations is represented by a set of continuous architecture weights.

• Method: The supernet weights and architecture weights are jointly optimized via standard gradient descent on a validation set.
• Efficiency: It's more efficient than reinforcement learning-based NAS but still requires substantial training of the supernet.
• Contrast with Proxies: DARTS performs a gradient-based search over architecture parameters. Zero-cost proxies are gradient-free and use analytical or one-pass metrics. They represent orthogonal approaches: DARTS for a continuous, differentiable search within a supernet; proxies for instant, training-free scoring of discrete candidates.