Neural Architecture Search (NAS)

A controller network (typically an RNN) is trained via reinforcement learning to generate architecture descriptions (actions). The performance of each sampled architecture on a validation set serves as the reward signal, which is used to update the controller's policy via policy gradient methods (e.g., REINFORCE).

• Key Paper: Zoph & Le (2017) - Neural Architecture Search with Reinforcement Learning.
• Pro: Can discover novel, high-performing architectures.
• Con: Extremely computationally expensive, often requiring thousands of GPU days.

This strategy treats architecture search as an evolutionary optimization problem. A population of candidate architectures (genotypes) is maintained. Architectures are selected based on fitness (validation accuracy), then mutated (e.g., add/remove layer, change operation) or crossed over to produce the next generation.

• Key Variants: Regularized Evolution, AmoebaNet.
• Pro: Highly parallelizable; good at exploring diverse regions of the search space.
• Con: Still computationally intensive; requires careful design of mutation operations and population management.

This method formulates the search space as a continuous supernet where the choice between candidate operations (e.g., 3x3 conv, 5x5 conv, identity) is relaxed using a continuous, learnable architecture weight (alpha). The supernet weights and architecture parameters are jointly optimized via standard gradient descent.

• Key Method: DARTS (Differentiable Architecture Search).
• Pro: Dramatically faster than RL/Evolutionary methods (orders of magnitude).
• Con: Prone to discretization gap (the derived discrete architecture often underperforms the continuous supernet); high memory consumption.

A supernet encompassing all possible architectures in the search space is trained once. During search, candidate sub-networks are evaluated by inheriting weights from the corresponding shared weights in the supernet, without separate training. The best-performing sub-network is then retrained from scratch.

• Key Methods: ENAS (Efficient NAS), Single-Path One-Shot.
• Pro: Highly efficient; search cost is amortized into one supernet training run.
• Con: Weight-sharing introduces bias; the performance ranking of sub-networks under shared weights may not correlate perfectly with their stand-alone performance (the ranking consistency problem).

Treats the mapping from architecture to validation performance as an expensive black-box function. A surrogate model (e.g., Gaussian Process) is used to model this function. An acquisition function (e.g., Expected Improvement) guides the selection of the most promising architecture to evaluate next, balancing exploration and exploitation.

• Pro: Sample-efficient; well-suited for very expensive evaluations.
• Con: Scaling to high-dimensional search spaces is challenging; performance depends heavily on the choice of kernel and acquisition function.

These are simpler, iterative methods that start from a baseline architecture and make local modifications to improve performance. Examples include hill climbing, where random mutations are accepted if they improve accuracy, and network morphism, which expands an existing network in a functionally preserving way.

• Use Case: Often used for fine-tuning or in constrained hardware-aware NAS pipelines.
• Pro: Simple, interpretable, and low overhead.
• Con: Can get stuck in local optima; less effective for discovering radically new architectures from scratch.

NAS is used to automatically discover neural network architectures that are optimized for hardware-specific constraints like latency, memory footprint, and power consumption. By incorporating these metrics directly into the search objective, NAS can produce models that outperform hand-designed counterparts for edge devices.

• Search Space: Defines possible layer types (e.g., depthwise convolutions, inverted residuals), connectivity patterns, and kernel sizes.
• Search Strategy: Uses reinforcement learning, evolutionary algorithms, or gradient-based methods to explore the space.
• Constraint Integration: The search is guided by a multi-objective reward function balancing accuracy with metrics like FLOPs or on-device latency.

This advanced application of NAS directly optimizes architectures for a specific target hardware platform (e.g., a mobile NPU, microcontroller, or GPU). It uses a performance predictor or direct on-device measurements to estimate latency/energy during the search, ensuring the final model is co-designed with the silicon.

• Performance Estimation: Employs lookup tables, neural predictors, or micro-benchmarks to avoid costly full training and deployment for every candidate.
• Platform Diversity: Can target diverse hardware from Google's Edge TPU and NVIDIA Jetson to ARM Cortex-M series MCUs.
• Result: Produces a Pareto-optimal frontier of models offering different trade-offs between accuracy and efficiency for the chosen hardware.

NAS is applied to find optimal compressed variants of large Transformer models (e.g., BERT, GPT) by searching over sub-components. This is more systematic than applying fixed compression recipes.

• Searchable Elements: Attention head count, hidden dimension size, number of layers, feed-forward network intermediate size, and activation functions.
• Techniques Combined: The search can inherently apply structured pruning, layer reduction, and efficient attention mechanism selection in a unified process.
• Outcome: Creates task-specific, compact Transformer models (e.g., TinyBERT, MobileBERT) that retain essential capabilities for deployment on servers or powerful edge devices.

When a new modality or task lacks established, efficient architectures, NAS can discover them from scratch. This is valuable for multi-modal or sensor-fusion tasks on edge devices where standard models may be suboptimal.

• Domain Examples: Designing efficient networks for radar signal processing, LiDAR point cloud classification, or audio-visual event detection.
• Process: The search space is constructed from fundamental operations relevant to the input data type (e.g., 1D convolutions for audio, 3D convolutions for video).
• Benefit: Discovers novel, high-performing layer combinations that human designers might not hypothesize, potentially achieving better accuracy-efficiency trade-offs.

NAS can be framed as a pruning problem. Instead of training a large network and then removing weights, the search directly seeks a high-performing sub-network within a large, over-parameterized supernet. This connects NAS to the Lottery Ticket Hypothesis.

• One-Shot NAS: A supernet encompassing all possible architectural choices is trained once. The search then evaluates sub-networks by inheriting weights from this supernet without retraining.
• Differentiable NAS (DNAS): Uses continuous relaxation of the architecture choices, allowing gradient-based optimization to learn both the network weights and the importance of each architectural component simultaneously.
• Advantage: More computationally efficient than traditional NAS methods and directly produces a sparse, optimal sub-architecture.

NAS can be combined with Quantization-Aware Training (QAT) to search for architectures that are inherently robust to precision reduction. This co-design approach yields models optimized for integer-only inference on edge hardware.

• Joint Search: The search process simulates quantization (e.g., to INT8) during candidate architecture evaluation.
• Objective: Maximizes accuracy under quantized inference, rather than full-precision accuracy. This discovers operations and activation functions that minimize quantization error.
• Deployment Ready: The resulting model is pre-adapted for frameworks like TensorFlow Lite or TensorRT, requiring minimal post-search calibration.

These are hand-designed neural network structures that prioritize low parameter counts and fast inference. NAS often searches within or builds upon these proven design patterns. Key examples include:

• MobileNet: Uses depthwise separable convolutions for efficient vision tasks.
• EfficientNet: Employs a compound scaling method to balance depth, width, and resolution.
• Transformer variants like MobileViT: Combine convolutional efficiency with the global receptive field of attention for mobile devices.

A specialized form of NAS where the search algorithm directly optimizes for metrics specific to the target deployment hardware. The search objective extends beyond accuracy to include:

• Latency: Measured directly on the device or via a proxy model.
• Memory Footprint: Constraining peak RAM usage.
• Energy Consumption: Estimating power draw for battery-operated devices.
• Compatibility: Ensuring operators are supported by the hardware's acceleration libraries (e.g., specific INT8 ops).

An efficient NAS paradigm where all candidate sub-architectures are encoded as parts of a single, over-parameterized supernetwork. This allows for massive search space exploration with minimal compute cost.

• Mechanism: The supernetwork is trained once. Architectures are evaluated by sampling and inheriting weights from the supernetwork, avoiding training each from scratch.
• Trade-off: While fast, weight-sharing can lead to inaccurate ranking of candidates due to co-adaptation of parameters within the supernet.

A gradient-based NAS method that formulates the search for discrete architectural components as a continuous, differentiable optimization problem.

• Core Idea: The choice between operations (e.g., 3x3 conv, 5x5 conv, skip-connect) is relaxed using a continuous mixture. Architecture parameters are learned via gradient descent alongside network weights.
• Advantage: Significantly faster than reinforcement learning or evolutionary-based searches.
• Limitation: The continuous relaxation can cause a disparity between the search model and the final discrete architecture, and it is memory-intensive.

The mathematical framework at the heart of practical NAS, where the goal is to find architectures that best satisfy multiple, often competing constraints. For edge AI, this is rarely just about top-1 accuracy.

• Pareto Frontier: NAS aims to discover architectures on the Pareto frontier—the set of solutions where improving one objective (e.g., accuracy) necessarily worsens another (e.g., latency).
• Common Objectives: Model Size (MB), Inference Latency (ms), Energy Use (Joules), and Accuracy (%).

A compression technique that removes redundant parameters from a trained network. NAS and pruning are complementary: NAS designs an efficient skeleton, while pruning further refines it.

• Structured Pruning: Removes entire neurons, filters, or layers. Results in a smaller, dense network well-suited for standard hardware. Often used as a constraint within NAS.
• Unstructured Pruning: Removes individual weights, creating a sparse model. Achieves high compression ratios but requires specialized software or hardware for speedups.
• The Lottery Ticket Hypothesis suggests that pruned, efficient subnetworks exist at initialization within larger models, a concept closely related to weight-sharing in One-Shot NAS.