ProxylessNAS

This is the core technique that makes direct search computationally feasible. Instead of evaluating all possible candidate operations (e.g., 3x3 conv, 5x5 conv, skip connect) in the over-parameterized network simultaneously, ProxylessNAS binarizes the architecture parameters. For each layer or block, it activates only one path (operation) at a time during the forward/backward pass, while keeping others dormant. This reduces memory and compute overhead from O(N) to O(1) per layer during search, allowing the search to be conducted directly on the large target dataset (e.g., ImageNet) and target hardware.

• Binary Gates: A set of architecture parameters α controls a binary gate for each operation. During training, the gate is sampled to be 0 or 1.
• Gradient Estimation: Since sampling is non-differentiable, gradients for the architecture parameters α are estimated using the straight-through estimator (Gumbel-Softmax trick), enabling end-to-end gradient-based optimization.

A defining feature of ProxylessNAS is its direct incorporation of hardware performance metrics into the search objective. Instead of using a proxy like FLOPs or parameter count, it measures and optimizes for real, on-device latency.

• Latency Look-Up Table (LUT): A pre-built table stores the measured latency of each candidate operation (e.g., 3x3 depthwise conv, 5x5 conv) on the target hardware (e.g., a specific mobile CPU, microcontroller).
• Differentiable Loss Term: The expected latency of the current architecture, computed as a weighted sum based on the architecture parameters α, is added as a regularization term to the standard cross-entropy loss: Loss = Loss_CE + λ * log(Latency). The gradient of this latency term guides the search towards faster operations.
• Hardware-Aware: This allows the discovered architecture to be Pareto-optimal for the specific chip's characteristics, such as memory bandwidth and supported instruction sets.

ProxylessNAS builds upon the gradient-based search paradigm of Differentiable Architecture Search (DARTS) but removes its major limitation: the need for a proxy. DARTS relaxes the discrete search space by representing the choice between operations as a continuous mixture (softmax over all options). While efficient, this requires maintaining all operations in memory, forcing the search to be done on a smaller proxy dataset (e.g., CIFAR-10) before transferring the cell to a larger network.

ProxylessNAS modifies this by:

• Replacing Continuous Mixtures with Binary Paths: Using binarization to break the memory bottleneck.
• Eliminating the Proxy Stage: Enabling direct search on the full task (ImageNet) and final network macro-architecture.
• Retaining Gradient-Based Optimization: Keeping the efficiency of using gradient descent to update architecture parameters, as opposed to slower reinforcement learning or evolutionary methods.

By using path-level binarization, ProxylessNAS achieves a memory footprint similar to training a single model, not a supernet containing all possible operations. This is critical for searching directly on large-scale tasks like ImageNet classification, which was infeasible for prior differentiable NAS methods.

Process Flow:

1. Construct an Over-parameterized Network: Define a network where each layer has N candidate operations.
2. Binarize and Sample: For each training batch, for each layer, sample a binary gate to select one operation path.
3. Forward/Backward Pass: Only the weights of the active path are used and updated. Compute loss and gradients.
4. Update Architecture Parameters: Use the straight-through gradient estimator to update the probabilities α for each operation.
5. Prune and Derive Final Architecture: After search, the operation with the highest α value at each layer is selected to form the final, compact network.

The principles of ProxylessNAS are directly applicable and highly valuable for Tiny Machine Learning and microcontroller deployment. Its hardware-aware, direct-search methodology is ideal for discovering models under extreme constraints.

• Direct MCU Latency Optimization: The latency LUT can be built by profiling kernels on a specific microcontroller (e.g., ARM Cortex-M4/M7), searching for architectures that maximize accuracy within a strict SRAM/Flash memory budget and inference time limit.
• Co-Design with Inference Engines: Frameworks like MCUNet demonstrate the next step: co-designing the neural architecture (via a TinyNAS search inspired by ProxylessNAS) with a tailored inference engine (TinyEngine) to push the limits of on-device vision.
• Search Space for Micro-DNNs: The candidate operations in the search space are tailored for MCUs, including depthwise separable convolutions, inverted residual blocks, and various kernel sizes, all with strict parameter and activation map size limits.

ProxylessNAS was proposed to address the significant shortcomings of proxy-based NAS methods, which introduced a gap between search and deployment conditions.

Proxy-Based NAS (e.g., Early NASNet, MNasNet Proxy):

• Proxy Task: Searches on a smaller dataset (CIFAR-10).
• Proxy Metric: Optimizes for FLOPs or parameters, not real latency.
• Two-Stage Process: 1. Search on proxy. 2. Scale up and retrain on target task. The optimal architecture on the proxy may not be optimal on the target.

ProxylessNAS:

• Target Task: Searches directly on the full dataset (ImageNet).
• Target Metric: Optimizes for measured on-device latency.
• One-Stage Process: Search and hardware-aware optimization happen concurrently, leading to architectures that are directly deployable. This closes the simulation-to-reality gap in neural architecture search.

Neural Architecture Search (NAS) is the overarching automated process for discovering optimal neural network architectures for a given task and computational budget. It treats the network design—choices of layers, connections, and operations—as a search problem. Common strategies include:

• Reinforcement Learning: Using an RNN controller to propose architectures and receive accuracy rewards.
• Evolutionary Algorithms: Applying genetic operations like mutation and crossover to a population of architectures.
• Gradient-Based Methods: Making the search space continuous and differentiable to optimize with gradient descent. ProxylessNAS is a specific, efficient instantiation of gradient-based NAS that eliminates proxy tasks.

Hardware-Aware Neural Architecture Search is a critical variant where the search algorithm directly optimizes for metrics beyond accuracy, specifically targeting the performance characteristics of the deployment hardware. Instead of just FLOPs, it incorporates direct measurements or predictors for:

• Inference Latency (on a specific MCU, CPU, or NPU)
• Energy Consumption
• Peak Memory Usage
• Model Size ProxylessNAS is inherently hardware-aware, as it uses a latency estimation model built from empirical measurements on the target device (e.g., a microcontroller) to guide the search, ensuring the final architecture is optimal for that specific silicon.

Differentiable Architecture Search (DARTS) is the foundational gradient-based method that enables efficient NAS. It formulates the search space as a supernet or over-parameterized network where every possible architectural choice (e.g., which operation to use between two nodes) is represented simultaneously. Each choice is weighted by a continuous, learnable architecture parameter (alpha).

• The supernet is trained by jointly optimizing the standard model weights (with gradient descent) and the architecture parameters (with gradient descent).
• After training, a discrete architecture is derived by selecting the choices with the highest architecture parameters. ProxylessNAS builds upon this but removes the memory-intensive requirement of maintaining all candidate operations in memory simultaneously through path-level binarization.

The Once-For-All (OFA) Network is a training and deployment paradigm designed for diverse hardware constraints. A single, large supernet is trained to contain a vast number of smaller sub-networks of varying depths, widths, and kernel sizes through weight sharing.

• After training, specialized sub-networks can be extracted for different target devices (e.g., a tiny model for a microcontroller, a larger one for a mobile phone) without any retraining.
• This enables efficient hardware-aware search by simply evaluating the latency/accuracy of many sub-networks from the pre-trained OFA network. While ProxylessNAS searches for one optimal architecture, OFA provides a pre-baked portfolio of many efficient architectures from one training run.

MCUNet represents the state-of-the-art in co-design for TinyML on microcontrollers. It doesn't just search for an efficient neural architecture; it jointly optimizes two components:

1. TinyNAS: A hardware-aware neural architecture search that operates under an extremely tight memory budget (e.g., < 256KB of SRAM).
2. TinyEngine: A memory-efficient inference library that generates tailored, in-place code to avoid memory peaks. The key innovation is the tight feedback loop: TinyNAS designs networks based on the actual memory profiling provided by TinyEngine. ProxylessNAS is a core search algorithm that can be used within a co-design framework like MCUNet to achieve record-breaking ImageNet accuracy on microcontrollers.

EfficientNet and MobileNet are families of hand-designed, highly efficient convolutional neural networks that established many best practices later automated by NAS.

• MobileNet introduced the widespread use of depthwise separable convolutions to drastically reduce computation.
• EfficientNet proposed a compound scaling method (simultaneously scaling depth, width, and resolution) for balanced accuracy/efficiency gains. These architectures serve as both strong baselines and common search spaces for NAS methods. ProxylessNAS often searches over blocks inspired by MobileNet (e.g., MobileNetV2's inverted residual block) to find novel, more efficient permutations than the original hand-designed versions. EfficientNet-Lite variants are further optimized for integer-only hardware like Edge TPUs.