Zero-Cost Proxy

The gradient norm measures the magnitude of the gradient of the loss with respect to the network's weights after a single mini-batch of data. A higher gradient norm at initialization suggests the network is in a region of parameter space with a steeper loss landscape, which is often correlated with faster convergence and better final performance. This proxy requires a single forward and backward pass.

• Mechanism: Calculated as the L1 or L2 norm of the gradients.
• Use Case: Initial screening in NAS to filter out architectures with vanishing gradients.
• Limitation: Sensitive to batch size and weight initialization scheme.

Synaptic saliency, formalized by the SNIP (Single-shot Network Pruning) score, estimates the importance of a connection (synapse) by approximating the effect of its removal on the loss. It computes the gradient of the loss with respect to each parameter, multiplied by the parameter's value. Connections with low saliency scores are candidates for pruning before any training occurs.

• Formula: Saliency = |θ * ∇_θ L|, where θ is the weight and ∇_θ L is its gradient.
• Advantage: Provides a parameter-level importance score, enabling fine-grained pruning.
• Application: Used for one-shot, pre-training pruning to create sparse subnetworks.

NASWOT (NAS Without Training) is a zero-cost proxy based on the similarity of activation patterns across inputs. It posits that architectures producing more diverse, uncorrelated activation maps for different inputs have greater expressive power. The proxy score is derived from the kernel matrix of activations using a linear kernel.

• Core Idea: Measures the rank of the activation matrix for a batch of data. Higher rank indicates less linear dependence between activation vectors.
• Cost: Requires only a single forward pass with a batch of data.
• Performance: Has shown strong correlation with final test accuracy on NAS benchmarks like NAS-Bench-101.

These metrics use curvature or gradient flow information. The Fisher Information estimates parameter importance based on the expected gradient outer product. GraSP (Gradient Signal Preservation) selects weights to prune based on their contribution to preserving the gradient flow through the network, aiming to maintain the directional decrease in loss.

• Fisher: Saliency ≈ (∇_θ L)². Used in magnitude-based pruning with a gradient-informed prior.
• GraSP: Scores parameters by how their removal would affect the gradient norm of the remaining weights. Targets pruning that minimizes disruption to optimization.
• Commonality: Both use gradient information from a single mini-batch to infer parameter importance for sparsity.

These are advanced, theoretically motivated zero-cost proxies. The Zen-Score is derived from the theory of Neural Tangent Kernel (NTK) and measures the trainability of a network at initialization. LogSynflow is a stable, scale-invariant variant of synaptic flow scores that avoids numerical instability by applying a logarithmic transformation.

• Zen-Score: Correlates the NTK eigenvalues with generalization. Calculated via forward passes of perturbed inputs.
• LogSynflow: Score = ∑ log(1 + |θ * ∇_θ L|). Designed to be robust across different architectures and initialization scales.
• Utility: Enable more reliable ranking of architectures across vastly different search spaces (e.g., CNNs vs. Transformers).

In Automated PEFT Configuration, zero-cost proxies are used to rapidly evaluate different adapter configurations, pruning masks, or low-rank dimensions without full fine-tuning. This automates the design of efficient architectures.

• Example: Using gradient norm or SNIP to decide which attention heads in a transformer to freeze versus adapt with LoRA.
• Example: Using NASWOT to select the optimal rank for LoRA matrices or the insertion points for adapter layers.
• Benefit: Reduces the configuration search for PEFT from days of training to minutes of profiling, making hypernetworks and architecture search for adaptation feasible.

Weight sharing is a core efficiency technique in NAS where a single set of parameters in a supernet is shared across many different sub-architectures. This allows for the approximate evaluation of candidate models without training each one independently. Zero-cost proxies leverage this shared parameterization to compute heuristic metrics.

• Function: Dramatically reduces the computational cost of evaluating architecture candidates.
• Foundation for Proxies: Enables the calculation of metrics like gradient flow or parameter saliency on the supernet, which are then used as proxies for the performance of individual subnets.
• Challenge: Requires careful supernet design to ensure shared weights are representative of standalone training.

A neural predictor is a machine learning model (e.g., a regressor) trained to predict the final performance of a neural network architecture from its structural encoding. While more accurate than zero-cost proxies, it requires a costly initial training phase. Proxies are often used to gather the initial data to train these predictors.

• Data Requirement: Needs a dataset of (architecture, performance) pairs, which can be expensively gathered.
• Proxy Synergy: Zero-cost proxies can generate a large, cheap dataset of (architecture, proxy_score) pairs to pre-train or warm-start the neural predictor, improving its sample efficiency.
• Accuracy vs. Cost: Offers higher prediction fidelity than heuristics but at a higher upfront computational cost.

This is the process of algorithmically determining the optimal sparsity pattern (which weights to prune) for a neural network. Zero-cost proxies, such as synaptic saliency (the expected change in loss from pruning a parameter), are directly used to score and rank parameters for pruning without any fine-tuning.

• Direct Application: Metrics like |weight * gradient| serve as a zero-cost proxy for a parameter's importance.
• Pruning Protocols: Used in magnitude-based pruning, gradient-based pruning, and early-phase pruning where networks are pruned at initialization.
• Benefit: Allows for the creation of sparse, efficient models in a fraction of the time required for iterative pruning and fine-tuning.