NAS Benchmark

The search space is the finite set of all possible neural network architectures contained within the benchmark. It is formally defined by:

• Operational primitives (e.g., 3x3 conv, 5x5 depthwise conv, max pooling).
• Connectivity patterns (e.g., chain, cell-based, DAGs).
• Architectural hyperparameters (e.g., number of layers, channel widths). A well-defined space ensures the benchmark is comprehensive yet tractable, covering architectures relevant to real-world tasks without being impossibly large.

This is the core lookup table mapping each architecture in the search space to its evaluated performance. Key metrics include:

• Task Performance: Final accuracy on validation/test sets (e.g., ImageNet top-1 accuracy, CIFAR-10 error rate).
• Hardware Efficiency: Measured latency, peak memory usage, energy consumption, and model size (parameters), often profiled on specific hardware (CPU, GPU, MCU).
• Training Trajectories: Some benchmarks store performance at intermediate epochs, enabling research on early stopping and performance prediction.

A strict, reproducible methodology for how each architecture's performance is measured. This eliminates variance and ensures fair comparisons. It specifies:

• Training recipe: Optimizer (SGD, Adam), learning rate schedule, batch size, number of epochs, and data augmentation.
• Hardware setup: The exact processor, memory, and software stack used for profiling latency and energy.
• Evaluation dataset: The specific train/validation/test splits used, preventing data leakage. Standardization here is critical; differing protocols make cross-benchmark comparisons invalid.

The specific machine learning problem the benchmark addresses. Common tasks include:

• Image Classification: The most common, using datasets like CIFAR-10, ImageNet, or specialized sets like ImageNet-Edge for TinyML.
• Natural Language Processing: E.g., benchmarks on the Penn Treebank for language modeling.
• Hardware-Specific Tasks: Such as keyword spotting (Speech Commands dataset) for microcontroller deployment. The chosen task and dataset directly influence the search space design and the relevance of the discovered architectures.

Mechanisms to ensure the benchmark data is accurate, consistent, and free from errors that could invalidate research. This involves:

• Statistical sanity checks: Verifying performance distributions are plausible.
• Reproducibility audits: Re-evaluating a random sample of architectures to confirm reported metrics.
• Cross-validation: Ensuring no architecture was unfairly advantaged by a lucky train/validation split. Without rigorous validation, a benchmark can mislead the entire research community, promoting algorithms that exploit measurement noise rather than true efficiency.

The search space defines the universe of all possible neural network architectures a NAS algorithm can explore. It is a critical, predefined constraint that determines the upper bound of what can be discovered.

• Parameterization: Defines allowable operations (e.g., conv3x3, depthwise-separable conv, pooling), layer connectivity patterns (e.g., chain, residual, DAG), and hyperparameters like channel counts.
• Impact on Benchmarking: A NAS benchmark's utility is tied to its search space. Benchmarks like NAS-Bench-101 use a cell-based search space, while NAS-Bench-201 uses a macro-architecture with variable operations per edge.
• Design Challenge: A space must be expressive enough to contain high-performing models but constrained enough for efficient search. Poorly designed spaces can render a benchmark irrelevant.

The search strategy is the core algorithm that navigates the search space to find high-performing architectures. It is the "how" of NAS, distinct from the "what" defined by the benchmark.

• Common Strategies: Includes Reinforcement Learning (RL), Evolutionary Algorithms, Bayesian Optimization, and Gradient-Based Methods like DARTS.
• Benchmark Interaction: NAS benchmarks provide a standardized testbed to compare the sample efficiency and optimality of different search strategies. A strategy's performance is measured by how quickly it finds architectures near the Pareto frontier in the benchmark dataset.
• Example: Researchers can claim "Strategy X finds a 94% accuracy model in 0.3 GPU days on NAS-Bench-201," enabling direct comparison to prior work.

A performance estimator is any method used to predict the final validation accuracy or hardware efficiency of a candidate architecture without fully training it from scratch. It is essential for accelerating NAS.

• Types: Ranges from low-fidelity estimators (training for few epochs on a subset of data) to zero-cost proxies (e.g., grad_norm, synflow) that use a single forward/backward pass, to learned predictors (neural networks trained on benchmark data).
• Role in Benchmarking: NAS benchmarks are the ground-truth datasets used to train and validate these estimators. The correlation between an estimator's prediction and the benchmark's true accuracy is a key metric of the estimator's quality.
• Hardware Estimation: For hardware-aware NAS, estimators predict latency, memory, or energy, often using a hardware cost model.

Weight sharing is a technique where a single, over-parameterized supernet encapsulates all architectures in the search space. Candidate models (sub-networks) inherit weights from the supernet, enabling rapid, low-cost evaluation.

• Mechanism: The supernet is trained once. To evaluate a sub-architecture, the corresponding path is activated without retraining.
• Connection to Benchmarks: This technique enabled the creation of one-shot NAS benchmarks (e.g., NAS-Bench-1Shot1). These benchmarks pre-train a supernet and provide the performance of sub-architectures using shared weights, which is faster but introduces approximation error compared to training each from scratch.
• Critical Consideration: Benchmark results from weight-sharing are not directly comparable to results from standalone training benchmarks, as they measure relative ranking within the supernet ecosystem.

A hardware cost model is a function that predicts the on-device performance metrics (latency, memory, energy) of a neural network architecture for a specific target hardware platform.

• Implementation: Can be a look-up table of measured operator costs, an analytical model, or a small neural network regressor.
• Use in Hardware-Aware NAS: The cost model is integrated into the search objective, allowing the algorithm to optimize for both accuracy and efficiency (e.g., Accuracy / (Latency)^λ).
• Benchmarking Role: For hardware-aware NAS benchmarks (e.g., FBNet), the cost model's predictions are part of the benchmark data. The accuracy of the cost model itself is paramount; inaccurate predictions lead to architectures that fail on real hardware. Some benchmarks include actual measured hardware metrics for validation.

Multi-Objective NAS optimizes for several, often competing, objectives simultaneously—most commonly accuracy vs. a hardware metric like latency or model size.

• Pareto Frontier: The set of architectures where no objective can be improved without degrading another. An architecture is Pareto-optimal if no other architecture is strictly better in all objectives.
• Benchmark Output: A comprehensive NAS benchmark provides the data to construct this frontier. It allows researchers to evaluate if their search algorithm can discover architectures on or near the true Pareto frontier of the benchmark.
• Evaluation Metric: Algorithms are compared using metrics like Hypervolume—the volume in objective space dominated by the discovered frontier—measured against the benchmark's ground-truth frontier.