ProxylessNAS

ProxylessNAS eliminates the proxy dataset (e.g., CIFAR-10) and proxy hardware (e.g., simplified latency lookup tables) used by prior methods. It directly searches and evaluates architectures on the target task (e.g., ImageNet) and target hardware device (e.g., a specific mobile phone or microcontroller). This is achieved by training a single over-parameterized network where architecture parameters are binarized during forward passes, allowing gradients to flow to only one path at a time, drastically reducing memory overhead. The search directly incorporates a hardware latency loss term, measured on-device, into the training objective.

To manage GPU memory during the direct search on large datasets, ProxylessNAS introduces a path-level binarization technique. Instead of maintaining all candidate operations in memory simultaneously (as in DARTS), it stores only one path at a time.

• During training, architecture parameters are binarized using a differentiable process.
• For each input batch, only the weights of the active binary path are activated and updated.
• This reduces GPU memory consumption to be equivalent to training a single model, not a massive supernet, enabling direct search on ImageNet-scale tasks.

ProxylessNAS uses gradient descent to optimize both the network weights and the architecture parameters, making it more efficient than reinforcement learning or evolutionary-based NAS. The architecture parameters (α) represent the probability of choosing between candidate operations (e.g., 3x3 conv, 5x5 conv, skip connect). The gradient with respect to these parameters is estimated using the Gumbel-Softmax trick or BinaryConnect, providing a differentiable approximation of the discrete architecture selection. This allows the search to converge significantly faster than black-box optimization methods.

A core objective is to discover models that meet strict real-time inference constraints. ProxylessNAS incorporates hardware feedback directly into the loss function: Loss = CrossEntropyLoss + λ * log(Latency). The latency term is not a crude estimate but is measured by building and profiling each candidate operation block on the actual target device (e.g., a mobile CPU/GPU or microcontroller) to create a per-operation latency lookup table. During search, the expected latency of the current architecture is computed and penalized, steering the search toward Pareto-optimal architectures for the target hardware.

The method demonstrates versatility by defining search spaces for both Convolutional Neural Networks (CNNs) for vision and Recurrent Neural Networks (RNNs) for language tasks. For CNNs, the space includes layer-level choices for operations (e.g., regular conv, depthwise separable conv, pooling) and connectivity. For RNNs, it searches for the best cell structure (the recurrent unit), including the type of activation functions and internal connections. This shows ProxylessNAS as a general framework for hardware-aware architecture discovery across domains.

ProxylessNAS can efficiently generate specialized architectures for different hardware backends from a single search run. Since the latency loss term is hardware-specific, running the search targeting a mobile GPU will yield a different optimal architecture than targeting a mobile CPU or a microcontroller (MCU). This is a key advantage over methods that search for a one-size-fits-all model, as it acknowledges that the computational efficiency of operations (e.g., depthwise convolutions vs. standard convolutions) varies dramatically across hardware platforms, leading to truly optimized deployments.

ProxylessNAS eliminates the proxy dataset and simplified training used by earlier NAS methods. Instead, it directly measures the inference latency of candidate architectures on the target hardware (e.g., a specific microcontroller or mobile CPU) during the search. This provides a ground-truth hardware cost model, ensuring the discovered network meets real-time performance constraints essential for TinyML deployment.

• Key Mechanism: Integrates a latency lookup table or a small regression model pre-profiled on the target device.
• Result: Architectures are optimized for the actual memory hierarchy, cache behavior, and instruction set of the deployment silicon.

A primary application is designing networks for kilobyte-scale memory budgets on microcontrollers. ProxylessNAS can incorporate peak RAM usage and flash memory footprint as direct constraints in its search objective.

• Search Parameters: It optimizes layer depth, channel width, and operator choice (e.g., favoring depthwise separable convolutions over standard convolutions) to minimize memory consumption.
• Outcome: Discovers models that fit within the SRAM and flash constraints of devices like the Arm Cortex-M series, enabling complex vision or audio models on sub-1MB devices.

ProxylessNAS can be extended to Quantization-Aware NAS (QA-NAS), where the search process evaluates architectures under simulated low-precision arithmetic (e.g., INT8). This discovers models inherently robust to the accuracy loss from post-training quantization.

• Process: The supernet is trained with quantization-aware training (QAT) techniques during the search.
• Benefit: Produces networks that maintain high accuracy after deployment with fixed-point inference on MCUs, avoiding the need for costly quantization-aware fine-tuning after architecture search.

The framework can search for architectures optimized across a Pareto frontier for multiple devices simultaneously. This is crucial for companies deploying a single model across a heterogeneous fleet of edge devices (e.g., different smartphone models or IoT sensors).

• Methodology: The search objective becomes a multi-task learning problem, balancing accuracy against latency on several target hardware platforms.
• Use Case: Enables "train once, deploy anywhere" for edge AI, extracting optimal sub-networks from a single supernet for each hardware variant.

ProxylessNAS provides a feedback loop for neural-hardware co-design. The ground-truth latency measurements can inform the design of next-generation neural processing units (NPUs) or microcontroller accelerators.

• Application: Hardware architects can use the search to identify common, efficient operator patterns (e.g., specific kernel sizes or activation functions) that should be hardwired into future silicon.
• Outcome: Creates a virtuous cycle where NAS discovers networks for efficient hardware, and new hardware is designed to accelerate the networks NAS finds.

ProxylessNAS has been successfully applied to discover state-of-the-art models for resource-constrained perception tasks. This includes:

• Keyword Spotting: Ultra-efficient audio models for always-on wake-word detection on microcontrollers.
• Visual Wake Words: Lightweight image classifiers for person detection in smart cameras, operating under severe power limits.
• Industrial Anomaly Detection: Compact networks for analyzing vibration or thermal sensor data on factory equipment.

These applications leverage the method's ability to directly trade off model accuracy, latency, and energy consumption for a specific sensor modality and hardware target.

Hardware-Aware Neural Architecture Search is a variant of NAS that incorporates hardware-specific constraints—such as latency, memory footprint, and energy consumption—directly into the search objective. Unlike standard NAS which only optimizes for accuracy, this method discovers models optimized for a specific deployment platform like a mobile phone or microcontroller.

• Key Objective: Find Pareto-optimal architectures that balance accuracy with real-world hardware efficiency.
• Common Constraints: Inference latency (ms), peak RAM usage (KB), and energy per inference (mJ).
• Search Integration: Constraints can be applied as hard limits, regularization terms in the loss function, or via a hardware cost model.

Differentiable Architecture Search (DARTS) is a gradient-based NAS method that formulates the discrete search space as a continuous, differentiable supernet. This allows architecture parameters (which represent the choice of operations) to be optimized via standard gradient descent alongside network weights.

• Core Innovation: Replaces reinforcement learning or evolutionary search with efficient gradient-based optimization.
• Supernet Training: All candidate operations in a layer are weighted by architecture parameters (alphas) during a bi-level optimization process.
• Relation to ProxylessNAS: ProxylessNAS builds upon DARTS by making the search proxyless and directly incorporating hardware latency, but it uses a similar continuous relaxation principle for efficient search.

One-Shot NAS is a family of methods that train a single, over-parameterized supernet once. This supernet contains all possible operations and pathways defined by the search space. Candidate sub-architectures are evaluated by inheriting weights from this shared supernet, eliminating the need for individual training.

• Weight Sharing: The supernet's parameters are shared across all sub-architectures, serving as a performance estimator.
• Efficiency Gain: Allows evaluation of thousands of architectures at the cost of training just one network.
• Foundational Concept: ProxylessNAS, DARTS, and Once-For-All (OFA) all utilize a one-shot supernet paradigm to make the search computationally feasible.

A hardware cost model in NAS is a predictive function that estimates the real-world efficiency metrics (latency, energy, memory) of a candidate neural network architecture on specific target hardware. It is a critical component for hardware-aware search without constant physical deployment.

• Implementation: Can be a pre-built lookup table (LUT), a small neural network regressor, or an analytical model.
• Use in Search: The cost model's prediction is integrated into the search objective, penalizing inefficient architectures.
• ProxylessNAS Application: ProxylessNAS uses a latency lookup table modeled for the target device (e.g., mobile GPU) to guide the gradient-based search toward faster architectures.

In Neural Architecture Search, the search space defines the universe of all possible neural network architectures the algorithm can explore. It is parameterized by choices of layer operations, connectivity patterns, and dimensions like channel count and kernel size.

• Key Dimensions:
  • Operator Search: Choice between conv3x3, depthwise conv, pooling, etc.
  • Channel Search: Optimizing the width (# of filters) per layer.
  • Kernel Size Search: Selecting spatial dimensions of convolutional filters.
  • Connectivity: Deciding how layers are connected (e.g., normal, residual, skip).
• Critical Impact: A well-designed search space constrains the search to feasible, high-potential architectures, improving efficiency and final model quality. ProxylessNAS typically operates within a mobile-friendly search space.

Multi-Objective Neural Architecture Search explicitly optimizes for multiple, often competing, objectives such as accuracy, latency, and model size. The goal is to discover a set of architectures on the Pareto frontier, where no objective can be improved without degrading another.

• Trade-off Exploration: Provides practitioners with multiple optimal models for different deployment scenarios.
• Search Strategies: Uses algorithms like evolutionary methods or gradient-based search with multi-objective loss functions.
• ProxylessNAS Context: ProxylessNAS formulates a latency constraint as part of a two-objective (accuracy + latency) optimization problem, aiming to find points on this Pareto frontier for a target device.