Once-For-All (OFA) Network

The core of OFA is the training of a single, over-parameterized supernet that encompasses a vast search space of potential subnet architectures. This is achieved through progressive shrinking, where the network is first trained as a large model, then progressively fine-tuned to support smaller subnets (e.g., with fewer channels, layers, or kernel sizes). This single training run encodes the knowledge for all contained subnets.

The supernet is designed with elastic, configurable dimensions that define the search space. Key elasticities include:

• Depth: Number of layers or blocks.
• Width: Number of channels in convolutional layers.
• Kernel Size: Choice of convolutional kernel (e.g., 3x3, 5x5, 7x7).
• Resolution: Input image size. At deployment, a specific subnet is instantiated by sampling a configuration from these elastic dimensions, creating a model tailored to specific latency or accuracy constraints.

The primary advantage of OFA is zero-shot specialization. After the supernet is trained once, it can instantly produce a specialized subnet for any target device (e.g., mobile phone, edge TPU) without any retraining or fine-tuning. A search algorithm evaluates candidate subnets from the supernet against a hardware performance lookup table or latency predictor to find the optimal architecture for the given platform's constraints.

To find the best subnet, OFA employs a hardware-in-the-loop search. This involves:

• A latency lookup table built by profiling key operators on the target hardware.
• An evolutionary search or other optimization algorithm that uses this latency model to navigate the architectural space. The objective is to maximize accuracy subject to a hard latency or FLOPs constraint, directly optimizing for the deployment environment's unique characteristics.

OFA decouples the expensive training cost from the search cost, unlike traditional Neural Architecture Search (NAS). In classic NAS, each candidate architecture requires training from scratch or via a weight-sharing proxy, making search prohibitively expensive. OFA performs one expensive training run for the supernet, after which searching through thousands of subnets is efficient and requires only inference, not training.

Applications: Mass production of models for diverse edge devices, rapid prototyping for new hardware, and efficient model serving in heterogeneous environments.

Limitations:

• Supernet Training Cost: The one-time training is more expensive than training a single model.
• Search Space Design: Performance is bounded by the predefined elastic dimensions.
• Potential Performance Gap: Subnets may underperform compared to models trained independently from scratch, a phenomenon known as the architecture gap.

OFA is a cornerstone technology for deploying AI to heterogeneous edge devices. A single OFA supernet, trained in the cloud, can be deployed to millions of devices—from smartphones to IoT sensors—where the optimal subnet is selected at runtime based on the device's specific hardware profile (e.g., CPU, NPU, memory). This eliminates the need to store and manage hundreds of separate model variants.

• Key Benefit: Enables a 'train once, deploy anywhere' workflow for product lines with diverse hardware specs.
• Example: A computer vision model for a smartphone product line can dynamically shrink or expand its depth/width to match the processing power of budget, mid-tier, and flagship devices, all from one base model.

In cloud inference services, OFA networks allow dynamic adjustment of model complexity to meet strict Service Level Agreements (SLAs) for latency. The serving system can select a faster, smaller subnet during peak traffic to maintain response times, and a larger, more accurate subnet during off-peak hours.

• Key Benefit: Provides a built-in performance knob for inference servers, optimizing the trade-off between accuracy and speed in real-time.
• Technical Mechanism: The subnet search is guided by a latency lookup table pre-measured on the target server hardware, ensuring predictions are hardware-accurate.

OFA provides the foundational supernet for many efficient Neural Architecture Search (NAS) workflows. Instead of training each candidate architecture from scratch—a prohibitively expensive process—NAS algorithms can evaluate thousands of subnetworks sampled from the pre-trained OFA supernet at near-zero cost.

• Key Benefit: Dramatically reduces the computational cost of Hardware-Aware NAS from thousands of GPU days to a few.
• Process: The supernet is trained with progressive shrinking, allowing subnetworks with different depths, widths, and kernel sizes to achieve high accuracy. The search then finds the Pareto-optimal subnet for a given hardware constraint.

OFA facilitates efficient federated learning and on-device personalization. Instead of fine-tuning a massive, fixed model on a user's device—which is often infeasible—the system can first select a suitable, efficient subnet from the OFA supernet. This smaller subnet is then fine-tuned locally on user data.

• Key Benefit: Reduces the computational and memory footprint of on-device learning, making personalization feasible on resource-constrained hardware.
• Privacy Advantage: Keeps user data on-device. Only the small, personalized subnet updates (or gradients) might be shared, aligning with federated learning principles.

OFA integrates seamlessly with other model compression techniques to create a comprehensive optimization pipeline. The supernet training can be combined with Quantization-Aware Training (QAT) and pruning to produce a search space of models that are not only sized differently but also quantized (e.g., INT8) and sparse.

• Key Benefit: Enables joint optimization across multiple compression dimensions (architecture, precision, sparsity) for extreme efficiency.
• Workflow: 1) Train a QAT-aware OFA supernet. 2) Search for the best subnet architecture. 3) Apply post-training pruning to the selected subnet. This yields a model highly optimized for a specific NPU's supported operations and memory hierarchy.

Beyond architecture dimensions, OFA can encode subnetworks optimized for different input resolutions. A vision supernet can contain subnets trained to process 224x224, 192x192, and 160x160 images. The system can choose a lower-resolution subnet for simple scenes (saving compute) and a higher-resolution one for complex scenes.

• Key Benefit: Enables input-adaptive computation, where the amount of compute is dynamically matched to the complexity of the input data.
• Use Case: A real-time video analytics system can process distant/ small objects with high resolution and nearby/ large objects with lower resolution, maintaining overall accuracy while reducing average latency.

Neural Architecture Search (NAS) is the automated process of designing optimal neural network architectures for a given task and hardware constraint. It uses search algorithms (e.g., reinforcement learning, evolutionary algorithms) to explore a vast design space. OFA is a specific, efficient implementation of NAS where a single supernet is trained to contain many architectures, enabling rapid, training-free deployment-time search.

• Key Distinction: While generic NAS often involves training each candidate architecture from scratch, OFA amortizes this cost by training once.

Hardware-Aware Neural Architecture Search is a variant of NAS that directly incorporates hardware-specific performance metrics as objectives or constraints during the search process. These metrics can include:

• Latency (measured on the target device)
• Power consumption
• Memory footprint
• Compute utilization

OFA is inherently hardware-aware, as its search space is designed to produce subnetworks with varying FLOPs, parameters, and latency profiles, allowing the selection of a subnet that meets precise hardware limits.

Knowledge Distillation is a model compression and training technique where a compact 'student' model is trained to mimic the behavior of a larger, more accurate 'teacher' model. The student learns not just from ground-truth labels but from the teacher's softened output probabilities (logits) or intermediate feature representations.

• Relation to OFA: Knowledge distillation can be used within the OFA training process. The large supernet can act as a teacher to guide the training of its contained subnetworks, improving their accuracy and stability without separate training runs.

Model Compression is an umbrella term for techniques that reduce the computational cost, memory footprint, and energy consumption of a neural network for efficient deployment. Key methods include:

• Pruning: Removing unimportant weights or neurons.
• Quantization: Reducing numerical precision of weights/activations (e.g., FP32 to INT8).
• Knowledge Distillation: Training a smaller model.
• Neural Architecture Search/OFA: Designing efficient architectures from the start.

OFA provides a structural compression solution, generating inherently smaller and faster subnetworks from a compressed architectural space.

Dynamic Neural Networks are models whose computational graph, architecture, or parameters can adapt at inference time based on the input or resource constraints. This includes:

• Early-exit networks: Simpler inputs exit through earlier layers.
• Conditional computation: Activating only parts of the network per input.
• OFA Networks: Selecting a different subnet architecture per deployment scenario.

OFA enables a form of instance-aware specialization, where the optimal subnet is chosen once per hardware platform, rather than per input.

Supernet Training is the foundational methodology behind OFA. It involves training a single, over-parameterized neural network (the supernet) that encompasses an enormous number of potential subnet architectures within its weight-sharing structure.

• Progressive Shrinking: The OFA paper introduced this specific training strategy, where the supernet is first trained as the largest subnet, then progressively fine-tuned to support smaller subnets (e.g., shallower depth, narrower width, lower resolution).
• Weight Sharing: All subnets share the weights of the supernet, which is the core innovation that enables the 'train once, deploy many' paradigm.