Once-For-All (OFA) Network

The OFA paradigm trains a single, over-parameterized supernet that encompasses a vast search space of possible sub-networks. This is achieved through progressive shrinking, where the network is first trained as the largest possible configuration. Smaller sub-networks (varying in depth, width, and kernel size) are then fine-tuned by inheriting and adapting weights from this larger parent network, eliminating the need to train each variant from scratch.

OFA introduces elasticity across three core architectural dimensions:

• Elastic Depth: The number of layers in a block can be dynamically adjusted.
• Elastic Width: The number of channels (filters) in a convolutional layer can be scaled.
• Elastic Kernel Size: The size of convolutional kernels (e.g., 3x3, 5x5, 7x7) can be selected per layer. This multi-dimensional elasticity allows for the generation of sub-networks optimized for specific trade-offs between accuracy, latency, and model size.

A key application of OFA is hardware-aware neural architecture search. After the supernet is trained, an evolutionary search or other optimization algorithm can be run on the target device (e.g., a specific mobile phone or edge GPU). The search evaluates thousands of sub-networks sampled from the supernet in seconds by weight sharing, identifying the architecture that delivers the best accuracy for a required latency or memory budget on that exact hardware.

OFA fundamentally decouples the expensive, one-time training cost from the efficient, many-time specialization cost. The supernet training is computationally intensive but performed only once. Subsequent architecture search for new deployment targets is extremely fast and cheap, as it involves only inference and weight sharing, not gradient-based training. This makes it practical to tailor models for a vast array of end-use scenarios.

OFA differs from classic Neural Architecture Search (NAS) methods. Traditional NAS often requires training each candidate architecture from scratch or with significant computational overhead. One-Shot NAS methods like OFA train a single supernet, enabling efficient ranking of sub-networks. This makes OFA a highly efficient weight-sharing NAS strategy that provides a ready-made portfolio of deployable models.

The OFA methodology was used to create the OFA Network and underpins the efficiency of models like EfficientNet. The principle of training a flexible supernet and then extracting optimally scaled sub-netways enables the creation of entire families of models (e.g., Tiny, Small, Medium, Large) from a single training run, ensuring consistent architectural principles and optimized performance across the efficiency spectrum.

Neural architecture search (NAS) is the automated process of discovering high-performing neural network architectures for a given task and dataset. It is the foundational technology that OFA builds upon. Instead of manually designing networks, NAS algorithms explore a search space of possible operations (e.g., convolution types, attention blocks) to optimize objectives like accuracy, latency, or model size. OFA implements a highly efficient form of NAS known as one-shot NAS.

A supernet is an over-parameterized neural network that encompasses many possible smaller sub-networks (architectures) within its structure. Weight sharing is the critical technique where all these sub-networks inherit their parameters from the single, shared supernet. This is the core innovation behind OFA: training one supernet once-for-all allows for the evaluation of thousands of derived architectures without retraining, as their performance is approximated by the shared weights.

Hardware-aware neural architecture search extends NAS by directly incorporating deployment constraints into the search objective. Instead of just maximizing accuracy, it optimizes for metrics like:

• Inference latency (measured on target CPU/GPU/TPU)
• Memory footprint
• Energy consumption OFA is a quintessential hardware-aware method. After training the supernet, it can rapidly search for the best sub-network that meets the precise latency or FLOPs requirements of a specific edge device, cloud instance, or mobile phone.

Differentiable neural architecture search (DNAS) is a gradient-based approach that makes the architecture search space continuous. It introduces architecture parameters (e.g., which operation to use in a layer) that are optimized via gradient descent alongside the model weights. OFA uses a form of progressive shrinking that can be viewed as a differentiable strategy. It starts training the full supernet and gradually shrinks the search space, allowing gradients to guide the specialization of sub-networks for different depth, width, and kernel size configurations.

In model deployment, engineers often face trade-offs between competing goals. Multi-objective optimization seeks solutions that balance these trade-offs, resulting in a Pareto frontier of optimal architectures. For example, a model cannot be simultaneously the smallest, fastest, and most accurate. OFA explicitly addresses this by enabling efficient search across the accuracy-latency or accuracy-model size Pareto frontier. From the one supernet, users can extract models optimized for different points on this frontier.

Model compression techniques like pruning, quantization, and knowledge distillation reduce a model's computational footprint. EfficientNets are a family of models discovered via NAS and scaled using compound scaling. OFA operates in a related but distinct space: it doesn't compress a single final model but rather trains a flexible supernet that can be elastically shrunk to produce already-efficient sub-models. It can be combined with post-training compression (e.g., quantizing an OFA-derived sub-network) for further gains on edge devices.