Once-for-All Network

A Once-for-All Network is fundamentally a trainable supernet—a large, over-parameterized neural network that encompasses a vast design space of potential sub-networks. This space is defined by elastic dimensions such as:

• Depth: Number of layers.
• Width: Number of channels per layer.
• Kernel size: Spatial dimensions of convolutional filters.
• Resolution: Input image size. During a single training phase, the supernet is trained with a progressive shrinking algorithm that samples these sub-networks, allowing them to share weights and learn collaboratively. This creates a single, versatile model repository.

OFA's core innovation is the decoupling of training from architecture search. The computationally expensive training process is performed only once. After this one-time training, the search for the optimal sub-network for a specific hardware target becomes an efficient evaluation process, not a retraining task. Practitioners can perform a zero-shot Neural Architecture Search (NAS) by querying the pre-trained supernet with different depth, width, and kernel configurations, instantly assessing their accuracy and latency on the target device (e.g., a specific ARM Cortex-M MCU). This reduces deployment time from weeks to minutes.

The primary goal of OFA is to produce sub-networks optimized for diverse edge hardware constraints. The search process directly optimizes for key deployment metrics:

• Latency: Measured directly on the target device or via a pre-built latency lookup table.
• Memory Footprint: Model size (Flash) and activation memory (RAM).
• Energy Consumption: Inferences per Joule (IPJ). This hardware-aware optimization ensures the extracted model is not just small, but is the most accurate model that can run within the strict milliwatt computing budget and memory limits of a microcontroller, avoiding the inefficiency of using a one-size-fits-all model.

The supernet is trained using a progressive shrinking algorithm. Training starts with the largest sub-network (full depth, width, kernel size). Once stable, the algorithm progressively fine-tunes the supernet to support smaller sub-networks, starting with easier dimensions like kernel size, then width, and finally depth. This sequential knowledge distillation ensures that smaller, nested sub-networks inherit performance from their larger counterparts. The result is a nested, weight-sharing hierarchy where all sub-networks achieve high accuracy without catastrophic forgetting of the larger architectures.

A single trained OFA supernet can service a heterogeneous fleet of edge devices. From a single artifact, you can extract:

• A tiny, 500KB model for a Cortex-M4 microcontroller performing visual wake words.
• A medium-sized model for a Cortex-M7 with DSP blocks.
• A larger, higher-accuracy model for an Ethos-U55 microNPU accelerator. This eliminates the need to train, maintain, and deploy separate models for each device type or performance tier. It simplifies Over-the-Air (OTA) updates and fleet management, as a device can request a model tailored to its current capabilities or battery state.

OFA is a pivotal Hardware-Aware Neural Architecture Search (HW-NAS) strategy. It provides the training methodology that enables efficient search. Frameworks like MCUNet build upon this concept by performing co-design—jointly searching the neural architecture (TinyNAS) and the underlying inference engine (TinyEngine) for microcontrollers. While OFA defines the searchable model space, systems like MCUNet optimize the full stack, including static memory allocation and operator fusion, to push performance boundaries on the most constrained devices. OFA is thus a core enabling technique for state-of-the-art TinyML.

The OFA supernet allows for elastic model scaling across multiple dimensions—depth, width, kernel size, and resolution. Practitioners can perform a zero-shot search within the trained supernet to instantly extract a sub-network that meets the exact latency, memory, and energy constraints of a target microcontroller (e.g., an ARM Cortex-M4 vs. M7) or microNPU (e.g., Arm Ethos-U55). This eliminates the need to train a unique model for every hardware variant in a product line.

In IoT deployments with thousands of heterogeneous devices, OFA provides a unified model source. A central supernet serves as a model repository. Different sub-networks can be deployed based on each device's:

• Available RAM/Flash (e.g., 256KB vs. 1MB)
• Compute capability (with/without DSP blocks)
• Power budget (milliwatt computing)
• Sensor quality (input resolution) This enables consistent application logic and accuracy scaling across an entire fleet from a single training cycle, simplifying Over-the-Air (OTA) updates and maintenance.

OFA enables dynamic adaptation to changing operational conditions on the edge. A device can switch between pre-extracted sub-networks at runtime to optimize for:

• Latency-critical modes: Use a smaller, faster sub-network when Worst-Case Execution Time (WCET) must be minimized.
• High-accuracy modes: Use a larger sub-network when power is abundant or decision confidence is paramount.
• Battery-saver modes: Use a tiny, ultra-efficient sub-network for extended operation. This is crucial for applications like always-on keyword spotting or visual wake words where the baseline model is minimal, but a more complex model activates on detection.

OFA is a practical enabler for Hardware-Aware Neural Architecture Search (HW-NAS) in TinyML. Instead of running prohibitively expensive search-and-train cycles for each hardware target, the supernet is trained once. The search cost is reduced to a few seconds of evaluation on a target device or simulator to profile the latency and memory of candidate sub-networks. This methodology underpins frameworks like MCUNet, which co-designs efficient models (via TinyNAS) and inference engines for microcontrollers.

The OFA design supports sub-networks that share feature extractors of varying capacities. This is exploited in multi-stage inference pipelines. For example, a visual system might use:

1. A tiny sub-network for initial anomaly detection or motion-triggered wake-up.
2. A medium sub-network for object presence classification (visual wake words).
3. The largest viable sub-network for fine-grained classification. All stages are derived from the same supernet, ensuring feature compatibility and eliminating the memory overhead of loading multiple independent models.

The OFA supernet serves as a standardized benchmarking testbed for the TinyML research community. It allows for fair, controlled comparisons of:

• Compiler optimizations (e.g., operator fusion, static memory allocation) across a continuum of model sizes.
• Hardware accelerator performance (e.g., DSP blocks, microNPUs) on identical algorithmic tasks.
• Novel compression techniques like post-training quantization, applied uniformly across sub-networks.
• Robustness studies to understand how accuracy degrades with model shrinkage across different architectures within the same family.

Neural Architecture Search (NAS) is an automated process for designing optimal neural network architectures. Instead of manual design, NAS uses algorithms (like reinforcement learning or evolutionary strategies) to explore a vast space of possible layer types, connections, and hyperparameters. In TinyML, NAS is constrained by hardware metrics like latency, memory footprint, and energy consumption to discover models that are both accurate and feasible to run on microcontrollers.

Hardware-Aware Neural Architecture Search (HW-NAS) is a specialized variant of NAS that directly incorporates target hardware performance as an objective. It goes beyond just model accuracy to optimize for:

• Inference Latency (measured on the actual device or a cycle-accurate simulator)
• Peak Memory Usage (for both RAM and Flash)
• Energy Consumption (Inferences Per Joule) This process is fundamental to the Once-for-All methodology, as the supernet is trained with explicit awareness of the diverse sub-networks' performance on target edge platforms.

Model compression is a suite of techniques used to reduce the size and computational cost of a neural network for deployment on resource-constrained devices. Key methods include:

• Quantization: Reducing numerical precision of weights/activations (e.g., FP32 to INT8).
• Pruning: Removing redundant weights or neurons.
• Knowledge Distillation: Training a small 'student' model to mimic a large 'teacher'. While Once-for-All networks generate efficient sub-networks through architectural search, these compression techniques can be applied further to the extracted models for maximum deployment efficiency on microcontrollers.

MCUNet is a pioneering, co-design framework for TinyML that demonstrates principles similar to Once-for-All. It jointly optimizes two components:

• TinyNAS: A hardware-aware neural architecture search that designs networks within a few hundred kilobytes of memory.
• TinyEngine: An efficient, memory-aware inference library that performs optimizations like in-place depth-wise convolution and static memory planning. This tight integration of the neural network architecture and the underlying inference runtime is critical for pushing the limits of what's possible on microcontrollers, much like the hardware-aware training of a Once-for-All supernet.

A supernet (or weight-sharing network) is a large, over-parameterized neural network that contains many smaller sub-networks within its structure. It is the foundational architecture used in Once-for-All and many NAS techniques.

• Weight Sharing: All sub-netways share the parameters (weights) of the larger supernet.
• Training Protocol: The supernet is trained once using techniques like progressive shrinking or sandwich rule sampling to ensure the shared weights are good for many different sub-network configurations (e.g., varying depths, widths, kernel sizes).
• Search & Extraction: After training, optimal sub-networks for specific hardware constraints can be extracted without retraining.

Tiny Machine Learning (TinyML) is the subfield of machine learning focused on developing and deploying ultra-low-power, memory-constrained models that can run inference directly on microcontrollers (MCUs) and other deeply embedded edge devices.

• Target Hardware: Devices with < 1 MB of RAM, often powered by batteries or energy harvesting.
• Key Constraints: Milliwatt power budgets, kilobyte-scale memory, and the need for deterministic execution.
• Applications: Keyword spotting, visual wake words, industrial anomaly detection. Once-for-All networks are a strategic enabler for TinyML, providing a method to source many efficient, hardware-tailored models from a single training process.