Operator Search

Operator search selects from a predefined set of candidate operations for each layer or node in the computational graph. This is a discrete optimization problem, as the algorithm must choose one operation (e.g., a 3x3 convolution) over another (e.g., a 5x5 depthwise convolution). Common operation sets include:

• Standard Convolutions (e.g., 3x3, 5x5)
• Depthwise Separable Convolutions (dramatically reduce FLOPs)
• Pooling Operations (Max, Average)
• Identity/Skip Connections (enable residual learning)
• Dilated Convolutions (increase receptive field)
• Grouped Convolutions The choice directly dictates the layer's parameter count, FLOPs, and memory access patterns.

Operator search is one of three primary axes in a typical NAS search space, working in concert with:

• Cell/Topology Search: Defines how nodes (layers) are connected.
• Channel/Skip Search: Determines the width (# of channels) and the use of residual connections. The search space is the Cartesian product of these choices. For example, a search space might allow choosing between 8 operations at each of 5 nodes within a cell, leading to 8^5 (32,768) possible operator configurations for that cell alone, before considering connectivity.

In Hardware-Aware NAS, operator search is guided by a multi-objective reward function that balances accuracy with hardware metrics. The search algorithm evaluates operators not just for accuracy but for their impact on:

• Inference Latency: Measured in milliseconds, often via a hardware cost model or on-device profiling.
• Peak Memory Usage: Critical for microcontrollers with KB of RAM.
• Energy Consumption: Measured in microjoules per inference.
• Model Size (Parameters): Affects flash storage. Operators like depthwise convolutions are frequently selected in hardware-aware searches due to their favorable FLOP-to-accuracy ratio.

The mechanism for selecting operators varies by NAS methodology:

• Reinforcement Learning (RL-NAS): A controller RNN samples a probability distribution over operations.
• Evolutionary Algorithms: Operations are encoded in a genome and mutated/crossed over.
• Differentiable Search (DARTS): The hardest problem. A continuous relaxation is applied: each edge is assigned a set of architecture parameters (alpha) for all candidate operations. A softmax over these alphas creates a weighted mixed operation. After search, a discretization step selects the operation with the highest alpha, pruning the others. This allows efficient gradient-based optimization of the operator choice.

The operator search results in a heterogeneous architecture where different layers use optimally selected, often different, operations. This contrasts with hand-designed networks which typically use repetitive blocks (e.g., all 3x3 convs). For example, a NAS-discovered network might sequence:

1. A standard 5x5 conv for early feature extraction.
2. A depthwise conv for efficient mid-level processing.
3. A dilated conv to capture context without downsampling.
4. A skip connection to preserve gradient flow. This heterogeneity is key to achieving superior efficiency-accuracy Pareto frontiers.

For microcontroller deployment, operator search incorporates extreme constraints:

• Fixed-Point Friendly Ops: Preference for operations (like ReLU) that quantize well vs. those that don't (like Swish).
• Kernel Size Constraints: Avoiding large kernels (7x7) that cause inefficient memory access on MCUs.
• Operator Fusion Awareness: Selecting sequences of operations (Conv + BatchNorm + ReLU) that can be fused into a single, efficient kernel during compilation.
• Flash Memory Footprint: The choice of operation affects the stored model size; complex operators require more code space. Searches may use MCU-in-the-loop profiling to get true latency/energy measurements for candidate operators.

The search space defines the universe of all possible neural network architectures an algorithm can explore. It is parameterized by fundamental building blocks:

• Node operations (e.g., convolution, pooling, identity)
• Connectivity patterns between nodes (e.g., sequential, residual, dense)
• Hyperparameters per layer (e.g., filter count, kernel size)

Operator search directly manipulates one axis of this space. A well-designed search space balances expressiveness (to find good models) with tractability (to make search feasible). Common patterns include cell-based spaces (searching for a repeating block) and hierarchical spaces.

The search strategy is the core algorithm that navigates the search space to find high-performing architectures. It defines how the search is conducted. Major strategy families include:

• Reinforcement Learning (RL): A controller network is trained to generate architectures, rewarded for high accuracy.
• Evolutionary Algorithms: A population of architectures is evolved using selection, mutation, and crossover.
• Gradient-Based Methods (e.g., DARTS): The search space is relaxed to be continuous, allowing architecture parameters to be optimized via gradient descent.
• Random Search & Bayesian Optimization: Used as efficient baselines. The strategy is decoupled from the search space; operator search can be integrated into any of these approaches.

A hardware cost model is a predictive function that estimates the real-world efficiency metrics of a candidate neural network architecture on specific target hardware. It is the bridge between the abstract search space and physical deployment constraints.

Types of Cost Models:

• Look-up Tables (LUTs): Pre-measured latency/energy for each operator on the target hardware. The model sums costs for a given graph.
• Analytical Models: Use roofline models or hardware performance equations.
• Neural Network Predictors: Small MLPs trained to predict latency or power from an architecture encoding.

These models allow the search algorithm (e.g., during operator selection) to optimize directly for metrics like inference latency, peak memory usage, or energy consumption without needing to deploy every candidate.

Differentiable Architecture Search (DARTS) is a seminal gradient-based NAS strategy that heavily influences modern operator search. It formulates the search space as a continuous supernet where each candidate connection between nodes is weighted by a continuous architecture parameter (alpha).

How it relates to Operator Search:

• The set of possible operations (conv3x3, conv5x5, pool, etc.) on an edge is treated as a mixed operation, weighted by these alpha parameters.
• The search jointly optimizes the supernet weights (via standard gradient descent) and the architecture parameters (via gradient descent on a validation set).
• After search, a discrete architecture is derived by selecting the operation with the highest alpha value on each edge. This provides an efficient, gradient-based method for differentiable operator selection.

Once-For-All (OFA) is a one-shot NAS and training paradigm designed for efficiency across diverse hardware constraints. It trains a single, large supernet that supports many sub-networks via weight sharing.

Its connection to Operator Search:

• The OFA search space includes multiple dimensions: depth, width, kernel size, and—critically—operators.
• The supernet is trained with progressive shrinking to ensure the shared weights are good for all sub-networks.
• After training, a specialized search (e.g., using a hardware cost model) is performed to extract the optimal sub-network for a given latency or memory target. This makes operator search a post-training selection step from a fixed, pre-trained set of possibilities, eliminating the cost of searching from scratch for each new constraint.

Microcontroller NAS (MCU-NAS) refers to hardware-aware neural architecture search methods specifically engineered for the extreme constraints of microcontroller units (MCUs). It represents the most stringent application of operator search.

Key Differentiators:

• Constraints: Targets kilobyte-scale SRAM/Flash and microjoule-level energy budgets.
• Cost Model Fidelity: Requires extremely accurate models for MCU-specific operations (e.g., depthwise convolution, fixed-point arithmetic).
• Search Space Design: Heavily pruned to exclude operators prohibitively expensive on MCUs (e.g., large dense layers, standard convolutions). Favors operators like depthwise-separable convolutions, pooling, and ReLU.
• Objective: Discovers models that are not just accurate, but are compilable and runnable within the tiny memory footprint and low clock speeds of devices like Arm Cortex-M series chips.