Multi-Objective Neural Architecture Search

The central output of a multi-objective optimization is not a single 'best' architecture, but a set of Pareto-optimal solutions. An architecture is Pareto-optimal if no other architecture is strictly better in all objectives. This creates a trade-off curve (or surface) where improving one metric (e.g., accuracy) necessitates sacrificing another (e.g., latency). Practitioners select from this frontier based on their specific application's priority weighting.

• Key Concept: Dominance. Architecture A dominates Architecture B if A is at least as good as B in all objectives and strictly better in at least one.
• Example: A frontier might show models ranging from 95% accuracy / 100ms latency to 98% accuracy / 300ms latency.

Multi-objective NAS explicitly defines and optimizes for several distinct, often conflicting, metrics. Common objectives include:

• Predictive Performance: Validation accuracy, F1-score, BLEU.
• Efficiency Metrics: Inference latency (ms), memory footprint (MB), number of parameters (M), FLOPs.
• Hardware-Specific Costs: Energy consumption (Joules), peak memory bandwidth utilization.
• Robustness Metrics: Adversarial robustness, calibration error.

The search algorithm must navigate this high-dimensional landscape where improving accuracy typically increases model size and latency.

Specialized algorithms are required to handle multiple objectives. Common approaches include:

• Scalarization: Combining objectives into a single weighted sum (e.g., Objective = Accuracy - λ * Latency). This simplifies the problem but requires pre-defining trade-off weights λ.
• Evolutionary/Multi-Objective Evolutionary Algorithms (MOEAs): Algorithms like NSGA-II (Non-dominated Sorting Genetic Algorithm II) maintain a population of architectures and use dominance-based ranking and crowding distance to evolve a diverse Pareto front.
• Bayesian Optimization with Multi-Objective Acquisition Functions: Uses a probabilistic surrogate model and acquisition functions like Expected Hypervolume Improvement (EHVI) to select the most informative architectures to evaluate next.

Accurate and efficient estimation of hardware metrics is critical. Since measuring latency or energy on real devices for thousands of candidates is prohibitive, multi-objective NAS employs:

• Performance Predictors: Lightweight neural networks or graph neural networks trained to predict latency/energy from an architecture encoding.
• Analytical Profilers: Look-up tables or analytical formulas that estimate cost based on operator type, data shape, and hardware characteristics.
• Zero-Cost Proxies: Heuristics like synaptic flow or gradient norm that correlate with final trained performance and can be computed in a single forward/backward pass, used for initial candidate screening.

These models allow the search to be guided by realistic deployment constraints from the outset.

Multi-objective search is typically built atop efficient NAS backbones to manage computational cost:

• Weight-Sharing Supernets: A single one-shot or differentiable supernet is trained. Candidate sub-architectures inherit weights, enabling rapid evaluation of both accuracy and efficiency proxies without full retraining.
• Once-For-All (OFA) Style: A supernet is trained to support many sub-net configurations (depth, width, kernel size). Multi-objective search then selects the optimal sub-net for a given hardware target from this pre-trained support set.
• Progressive Shrinking: The search may start with larger, more accurate models and progressively shrink them, evaluating the accuracy/efficiency trade-off at each step.

In the context of Parameter-Efficient Fine-Tuning, multi-objective NAS automates the design of adaptation modules. It can search for optimal:

• Adapter Architectures: The placement, dimension, and internal structure of adapter layers.
• LoRA Configurations: The rank r and target modules for Low-Rank Adaptation matrices.
• Sparse Fine-Tuning Masks: Which specific parameters in a frozen pre-trained model should be selectively updated.

The objectives balance task performance gain against the additional parameters, memory overhead, and inference slowdown introduced by the PEFT method, ensuring the adaptation is truly efficient for the target domain.

This is the most common multi-objective trade-off, optimizing for both high task performance (e.g., classification accuracy) and low inference latency. The search aims to find architectures on the Pareto frontier, where any improvement in latency would degrade accuracy, and vice versa.

• Use Case: Real-time applications like autonomous vehicle perception, video analytics, and live translation services.
• Search Strategy: Often uses a hardware-aware NAS approach, incorporating measured or predicted latency from a surrogate model or on-target hardware (e.g., a specific GPU or mobile processor).
• Example: Discovering a convolutional block that uses efficient operations like depthwise separable convolutions to maintain high ImageNet accuracy while reducing inference time by 40% on a mobile CPU.

This objective pair targets memory-constrained environments, seeking compact models that do not sacrifice excessive accuracy. It directly impacts storage and RAM requirements.

• Use Case: Deployment on edge devices, smartphones, and browsers where model download size and memory footprint are limiting factors.
• Mechanism: The search space is designed with operations that have a low parameter count. The objective function penalizes architectures based on their total number of trainable parameters or memory footprint.
• Outcome: A family of models like EfficientNet, discovered via NAS and compound scaling, which achieve state-of-the-art accuracy with an order-of-magnitude fewer parameters than previous models.

Optimizing for computational energy consumption is critical for battery-powered devices and large-scale data centers aiming to reduce operational costs and carbon footprint.

• Use Case: IoT sensors, always-on mobile applications, and environmentally conscious cloud inference clusters.
• Measurement: Energy consumption can be estimated via proxy metrics (e.g., FLOPs, memory access cost) or directly measured using hardware performance counters.
• Search Integration: The NAS algorithm treats energy as a primary objective, potentially favoring architectures with sparser activations, lower precision operations, or conditional computation pathways that activate only necessary components per input.

Here, the objective is to discover a single neural architecture that performs well across multiple, distinct tasks or datasets simultaneously, promoting inherent robustness and generalization.

• Use Case: Foundational models, vision-language models, and systems requiring a unified backbone for several downstream applications (e.g., object detection, segmentation, and classification).
• Methodology: The search evaluates candidate architectures by measuring their performance after a quick adaptation (e.g., via PEFT methods) on a held-out set of diverse tasks. The objective function aggregates these scores.
• Benefit: Reduces the need to search for and maintain a specialized architecture for each individual task, lowering long-term MLOps complexity.

This objective pairs standard accuracy with metrics of model robustness, such as accuracy under adversarial attacks or performance on out-of-distribution data. The goal is to find inherently more secure and reliable architectures.

• Use Case: Security-critical systems (fraud detection, cybersecurity), safety-critical applications (medical diagnosis, robotics), and any deployment where input data may be noisy or maliciously crafted.
• Search Process: During architecture evaluation, the candidate model is tested not only on a clean validation set but also on perturbed inputs (e.g., via Projected Gradient Descent attacks) or a corrupted validation set. The robustness score is fed into the multi-objective optimizer.
• Result: Architectures may develop structural properties, like smoother decision boundaries, that are more resilient by design.

A practical use case is using multi-objective NAS to generate a spectrum of optimal models for a diverse hardware portfolio from a single search, a concept central to the Once-For-All (OFA) network paradigm.

• Use Case: A product line spanning cloud TPUs, desktop GPUs, mobile phones, and microcontrollers, each with different compute, memory, and latency constraints.
• Process: The supernet is trained with weight-sharing. The multi-objective search is then run multiple times, each with a different constraint profile (e.g., max_latency: 5ms for mobile, max_latency: 50ms for cloud).
• Outcome: A single trained supernet can yield specialized sub-networks for each deployment target without retraining, dramatically simplifying the production pipeline.

Neural Architecture Search (NAS) is the automated process of discovering high-performing neural network architectures for a given dataset and task. It treats the model's design—layer types, connections, and hyperparameters—as a search problem.

• Core Methods: Includes reinforcement learning-based controllers, evolutionary algorithms, and gradient-based approaches like Differentiable NAS (DNAS).
• Objective: Traditionally focused on maximizing a single metric, such as validation accuracy. Multi-objective NAS extends this to balance competing goals like accuracy, latency, and model size.
• Key Innovation: Automates a process historically requiring extensive expert knowledge and trial-and-error.

Hardware-Aware NAS is a specialized form of architecture search that directly incorporates hardware performance metrics into the optimization objective. It is a critical precursor and component of multi-objective NAS.

• Optimization Targets: Searches for architectures that optimize for metrics like inference latency, memory footprint, energy consumption, or FLOPs on specific hardware (e.g., mobile CPUs, GPUs, or NPUs).
• Measurement: Uses direct on-device profiling, lookup tables (LUTs), or accurate surrogate predictors to estimate hardware performance without full training and deployment for each candidate.
• Outcome: Produces models that are not just accurate but are practically deployable within real-world system constraints.

Pareto Optimality is a fundamental concept in multi-objective optimization, defining the set of optimal trade-offs between competing objectives. In multi-objective NAS, the result is a Pareto frontier of architectures.

• Pareto Frontier: A collection of architectures where improving one objective (e.g., accuracy) necessarily worsens another (e.g., latency). No single "best" architecture exists, only a set of optimal compromises.
• Decision-Making: The frontier allows engineers to select an architecture based on deployment priorities (e.g., choose the fastest model within a 1% accuracy drop).
• Search Goal: Multi-objective NAS algorithms aim to efficiently discover and characterize this frontier, rather than converge to a single point.

The Once-For-All (OFA) network is a training and deployment paradigm that decouples training from search. A single supernet is trained to support a vast array of sub-networks, enabling efficient multi-objective search post-training.

• Supernet Training: A large, over-parameterized network is trained with techniques like weight sharing and progressive shrinking to ensure many sub-architectures (varying in depth, width, kernel size) perform well.
• Efficient Search: After training, the optimal sub-network for any given latency/accuracy trade-off can be found via evolutionary search or gradient methods in minutes, without any retraining.
• Use Case: Directly enables fast, hardware-aware multi-objective NAS from a single pre-trained asset, ideal for deploying models across diverse edge devices.

Differentiable Neural Architecture Search (DNAS) formulates the NAS search space as a continuous, differentiable optimization problem. It is a highly efficient method often used within multi-objective frameworks.

• Mechanism: Architectures are represented as mixtures of operations with continuous architecture parameters (e.g., softmax over candidate layers). Both model weights and these architecture parameters are optimized via gradient descent.
• Multi-Objective Extension: Can be adapted for multi-objective search by incorporating differentiable proxies for latency or FLOPs into the loss function (e.g., using a latency prediction model).
• Advantage: Significantly faster than reinforcement learning or evolutionary-based NAS, making iterative multi-objective exploration more feasible.

Zero-cost proxies are heuristic metrics that estimate the quality or efficiency of a neural network architecture without requiring any training. They are used for ultra-fast initial screening in multi-objective NAS.

• Purpose: Prune the search space by quickly ranking thousands of candidate architectures before committing resources to training the most promising ones.
• Examples: Metrics like gradient norm, synaptic saliency (SNIP), Fisher information, or jacobian covariance, computed using one or a few mini-batches of data.
• Multi-Objective Use: Proxies can be defined for both performance (trainability) and efficiency (e.g., parameter count, estimated FLOPs), allowing for rapid Pareto frontier approximation.