Neural Architecture Search (NAS)

The search space defines the universe of all possible neural network architectures the NAS algorithm can explore. It is a constrained design space that balances flexibility with tractability.

• Cell-Based Spaces: Define a repeating computational block (cell); the search finds the optimal internal connections and operations. The final architecture stacks these discovered cells. This is highly efficient and the most common approach.
• Macro Spaces: Search over the entire global structure of the network, including the number of layers, layer types, and connectivity. This offers maximum flexibility but has a vastly larger search space.
• Hierarchical Spaces: Combine both, searching over high-level structures built from lower-level discovered cells.

Key constraint: The search space must be large enough to contain high-performing models but small enough for the search to be completed in a reasonable time.

The search strategy is the algorithm that explores the search space to identify promising architectures. It dictates how the NAS system navigates the vast combinatorial possibilities.

• Reinforcement Learning (RL): Uses an RNN controller to generate architecture descriptions. The performance (reward) of a sampled architecture is used to update the controller policy. Pioneering but computationally expensive.
• Evolutionary Algorithms: Maintains a population of architectures, applying mutation (e.g., change an operation) and crossover to generate new candidates. Selection is based on fitness (accuracy). Robust but can be slow.
• Bayesian Optimization: Builds a probabilistic surrogate model (e.g., Gaussian Process) to predict architecture performance based on past evaluations. Actively selects the most promising architectures to evaluate next. Sample-efficient.
• Gradient-Based Methods: (e.g., DARTS) Formulates the search as a bi-level optimization problem. The architecture is represented by continuous relaxation of choices, allowing the use of standard gradient descent. Much faster but can suffer from discretization error.

The performance estimation strategy is the method used to evaluate, or estimate, the performance (e.g., accuracy, latency) of a candidate architecture sampled by the search strategy. This is the major computational bottleneck of NAS.

• Full Training & Validation: The gold standard. Each candidate architecture is trained from scratch on the training set and evaluated on the validation set. This is prohibitively expensive for large search spaces.
• Low-Fidelity Estimates: Use proxies to reduce cost:
  • Training for fewer epochs (e.g., 5 instead of 200).
  • Training on a subset of data or with lower resolution images.
  • Using a smaller model (fewer channels/filters).
• Weight Sharing / One-Shot Models: Train a single, over-parameterized supernet that encompasses all architectures in the search space. Candidate architectures are evaluated as subnetworks of this supernet using shared weights, eliminating individual training. This is the cornerstone of modern, efficient NAS (e.g., ENAS, DARTS, OFA).
• Predictor-Based: Train a separate model (a predictor) to map an architecture's encoding directly to its predicted performance after seeing many architecture-accuracy pairs.

Hardware-Aware NAS is a critical variant where the performance estimation strategy includes metrics beyond accuracy, such as latency, power consumption, or memory footprint on a target device (e.g., mobile phone, NPU).

• Multi-Objective Search: The search optimizes for a joint objective, often formulated as a weighted sum: Objective = Accuracy + λ * (1 / Latency). The trade-off parameter λ controls the balance.
• Latency Prediction: Instead of measuring latency on-device for every candidate—which is slow—a small latency lookup table or a latency predictor model is built. Each operation (e.g., 3x3 depthwise conv on a specific hardware) has a pre-measured cost; total model latency is summed from its parts.
• Platform-Specific Optimizations: The search space can be constrained to only include operators that are highly efficient on the target hardware (e.g., preferring depthwise separable convolutions for mobile, or specific activation functions for a given NPU).

Example: Google's MNasNet directly incorporated a real-world latency term into its reward function, discovering models that were both accurate and fast on Pixel phones.

The Once-For-All (OFA) paradigm and supernet training represent the state-of-the-art in efficient performance estimation and deployment flexibility.

• Supernet: A single, large neural network whose architecture is designed to contain an enormous number of smaller subnetworks within it (e.g., by having switchable choices for kernel size, depth, width).
• Progressive Shrinking: The OFA training process involves:
  1. Training the full supernet.
  2. Finetuning it to maintain accuracy while supporting smaller subnetworks (reduced depth).
  3. Further finetuning for even smaller subnetworks (reduced width, kernel choices).
• Deployment-Time Search: After the supernet is trained once, you can search for the optimal subnetwork for any given hardware constraint (latency, model size) in seconds, without any retraining. The selected subnetwork inherits the pre-trained weights.

This decouples the massive cost of training from the search process, enabling rapid hardware-specific specialization from a single trained model asset.

NAS Benchmarks are curated datasets and evaluation frameworks created to standardize the comparison of different NAS algorithms, addressing the field's reproducibility crisis.

• NAS-Bench-101: The first major benchmark. Contains the exact training/validation/test accuracy for 423k unique convolutional cell architectures on CIFAR-10. Allows researchers to query the performance of any architecture in the space instantly, eliminating training noise and cost.
• NAS-Bench-201 & NATS-Bench: Extend to more operations and include results on multiple datasets (CIFAR-10, CIFAR-100, ImageNet-16-120). Provide a more comprehensive evaluation.
• HW-NAS-Bench: Includes not just accuracy but also hardware metrics like latency and energy consumption for architectures on multiple real devices (e.g., FPGA, Raspberry Pi, Pixel).

These benchmarks allow for fair, low-cost comparison of search strategies by providing a ground-truth lookup table for the search space, focusing research on the efficiency of the search algorithm itself.

NAS reduces the time and expert knowledge required to develop a high-performing model for a novel problem domain. Instead of manual architecture engineering and trial-and-error, the search algorithm explores the space autonomously.

• Use Case: Rapid prototyping for new sensor modalities (e.g., radar, lidar), emerging data types, or niche industrial applications where proven architectures don't exist.
• Methodology: The search is guided by a task-specific loss function and validation dataset. Techniques like transferable NAS search for a network cell on a proxy task (like CIFAR-10) that generalizes well when stacked for a larger, more complex target task.

Real-world deployment requires balancing multiple, often competing, objectives. Multi-objective NAS searches for architectures along a Pareto frontier, finding the optimal trade-offs between key metrics.

• Common Objective Triad: Accuracy vs. Latency vs. Model Size.
• Enterprise Application: Allows CTOs and ML engineers to select a model from the Pareto-optimal set that precisely matches their production constraints—for example, choosing the fastest model under 10ms latency, or the smallest model with >95% accuracy for cost-effective scaling.

Beyond standard accuracy, NAS can optimize for robustness to adversarial attacks, input corruptions, or distribution shift. The search incorporates specialized loss functions or adversarial training loops during candidate evaluation.

• Mechanism: The fitness function evaluates a model's performance not just on a clean validation set, but also on perturbed or out-of-distribution data.
• Outcome: This can discover architectural inductive biases (e.g., specific activation functions or connectivity patterns) that inherently improve model stability and security, complementing traditional training-time defenses.

A specialized variant of NAS that directly incorporates hardware-specific performance metrics as objectives or constraints during the architecture search. Instead of just optimizing for accuracy, the search algorithm also targets metrics like:

• Latency (inference time on the target chip)
• Power consumption
• Memory usage
• Energy efficiency This ensures the final discovered architecture is not only accurate but also practical for deployment on the intended NPU, GPU, or mobile SoC. Search strategies often use predictors or proxies to estimate these hardware costs without full model deployment for every candidate.

A training paradigm that creates a single, large supernet encompassing a vast number of smaller subnetworks with different depths, widths, and kernel sizes. A key enabler for efficient NAS, it allows:

• One-time training of the supernet.
• Zero-shot deployment by searching the supernet for a subnet that meets specific hardware constraints (e.g., <5ms latency) without any retraining.
• Trade-off exploration across the accuracy-latency spectrum from a single model. This drastically reduces the computational cost of NAS by decoupling the expensive training process from the architecture search phase.

A model compression technique where a small, efficient student model is trained to mimic the behavior of a larger, more accurate teacher model. In the context of NAS:

• NAS can be used to discover the student architecture itself.
• The teacher's soft labels (probability distributions) provide a richer training signal than hard labels, often leading to better student performance.
• This allows NAS to find very small, hardware-optimized architectures that retain much of the performance of large, cumbersome models. It's a complementary technique often used in the final stage of training a NAS-derived model.

A model compression technique that removes redundant or less important parameters (weights) or entire neurons from a neural network. It interacts with NAS in two primary ways:

• As a post-processing step: A discovered architecture can be further compressed via pruning after training.
• As a search space primitive: The NAS search space can include pruned structures (e.g., channels to keep/remove) as part of the architecture definition. This allows the search to directly optimize for sparsity, which is highly beneficial for hardware that supports sparse computation, reducing both model size and inference latency.

A model optimization technique that simulates the effects of lower numerical precision (e.g., INT8) during the training process. For NAS, QAT is critical for hardware deployment because:

• It ensures the final NAS-discovered architecture is robust to quantization, preventing significant accuracy drops when the model is deployed with integer math on NPUs/TPUs.
• Hardware-aware NAS can use quantized latency as a more accurate cost metric during the search.
• The search can explore mixed-precision policies, assigning different numerical precisions (FP16, INT8) to different parts of the network to optimize the accuracy-efficiency trade-off.

The process of transforming a high-level neural network computational graph into an optimized, hardware-specific sequence of low-level operations. This is the essential bridge between a NAS-discovered architecture and efficient NPU execution. Key steps include:

• Operator Fusion: Combining ops (Conv + Bias + ReLU) into a single kernel to minimize memory traffic.
• Layout Optimization: Changing tensor data layouts (e.g., NCHW to NHWC) to match hardware preferences.
• Kernel Selection: Choosing the most efficient low-level implementation for each operation on the target NPU. A NAS algorithm that is aware of the compiler's capabilities can discover architectures that are not only theoretically efficient but also compile to highly optimized binaries.