Latency Constraint (NAS)

Latency constraints are defined as either hard or soft bounds. A hard constraint acts as a strict upper limit (e.g., < 10 ms); any candidate architecture exceeding this limit is invalid and discarded. A soft constraint is incorporated into the search objective as a penalty term, allowing the algorithm to explore architectures that slightly exceed the target if they offer significant accuracy gains, ultimately finding a better trade-off on the Pareto frontier.

Latency is not an abstract metric; it is intrinsically tied to the target hardware platform. The same neural network will have different inference times on a CPU, GPU, NPU, or microcontroller. Effective NAS integrates a hardware cost model—often a lookup table of operator latencies or a small predictor network—to estimate runtime without deploying every candidate. The most accurate method is hardware-in-the-loop search, where architectures are profiled on the actual device.

The latency constraint must be formalized into the search algorithm's optimization goal. Common formulations include:

• Multi-objective optimization: Treating accuracy and latency as separate objectives to find Pareto-optimal architectures.
• Weighted sum: Combining accuracy and a latency penalty into a single reward score: Reward = Accuracy - λ * Latency.
• Constraint-based search: Using methods like reinforcement learning where the agent is rewarded for accuracy but only if latency is below a threshold.

A latency budget directly shapes the discovered neural network topology. To reduce latency, NAS algorithms will favor:

• Depthwise separable convolutions over standard convolutions.
• Smaller kernel sizes (3x3 vs. 5x5).
• Reduced network width (fewer channels per layer).
• Shallower network depth.
• Efficient activation functions like ReLU6 over more complex ones.
• The search space itself (operator search, channel search) is designed to exclude operations known to be slow on the target hardware.

Constraints can be static or dynamic. A static constraint is a fixed target (e.g., 5 ms for a always-on audio keyword spotter). A dynamic constraint may vary based on system state, such as available battery level or processor load, leading to NAS searching for a family of models (e.g., via Once-For-All) that can be dynamically selected at runtime to meet changing latency budgets without redeployment.

Predicted latency from a cost model must be validated. Final candidate architectures undergo rigorous profiling on the target hardware to obtain ground-truth measurements. This accounts for real-world factors like memory bandwidth bottlenecks, cache effects, and operating system overhead. This step is crucial for Microcontroller NAS (MCU-NAS), where limited memory can cause unexpected swap delays, making pure FLOP-based estimates inaccurate.

Latency constraints are paramount for keyword spotting, voice activity detection, and audio event classification on microcontrollers. Models must process streaming audio buffers (e.g., 1-second windows) within the buffer period to enable continuous listening without gaps.

• Example: A wake-word detector on a smart home device must infer in < 100ms to feel instantaneous to the user.
• NAS Role: The search algorithm evaluates candidate architectures against a hardware cost model for the target MCU's CPU/DSP, pruning any design whose predicted inference time exceeds the audio frame deadline.

Vibration and anomaly detection models on motor controllers must analyze sensor data in real-time to trigger immediate shutdowns and prevent catastrophic failure. Inference latency directly correlates with reaction time and potential cost savings.

• Constraint: Latency is bounded by the physical system's time constants (e.g., a bearing failure may develop over milliseconds).
• NAS Approach: Uses hardware-in-the-loop search to profile candidates on the actual industrial-grade microcontroller, ensuring the selected model's inference loop fits within the critical sampling interval of high-frequency sensors.

Applications like person detection, hand gesture recognition, and visual anomaly inspection require frame-rate matching. For a 30 FPS video stream, total inference must complete in < 33ms per frame.

• Challenge: Latency includes sensor readout, image preprocessing, model inference, and post-processing.
• NAS Optimization: The search space is pruned to architectures using efficient operators like depthwise separable convolutions and optimized activation functions. The performance estimator predicts end-to-end latency, not just FLOPs, accounting for memory bandwidth bottlenecks on the target hardware.

For navigation and obstacle avoidance, LiDAR/vision-based perception models must deliver predictions within the robot's control loop cycle (often 10-100Hz). High latency causes instability or collisions.

• Multi-Objective Trade-off: NAS must balance latency with accuracy; a slightly less accurate model that is 2x faster may be safer.
• Methodology: Employs Pareto frontier search to find architectures that satisfy the hard real-time constraint while maximizing detection accuracy. The search may use evolutionary NAS to explore radically different backbones suited to the sensor fusion pipeline.

Health monitoring wearables (e.g., for heart arrhythmia detection or fall detection) run on coin-cell batteries and must provide continuous, low-latency analysis to enable timely alerts.

• Power-Latency Coupling: Lower latency often allows the CPU to return to sleep faster, reducing average power consumption.
• NAS Strategy: Implements a joint optimization of latency and energy consumption. The hardware cost model incorporates the target MCU's power states, favoring architectures that minimize active inference time to extend battery life.

In embedded control loops (e.g., drone stabilization, motor control), a TinyML model may provide a state estimate or high-level command. Its latency adds directly to the system's phase margin, affecting stability.

• Hard Requirement: Latency must be deterministic and bounded (e.g., < 5ms) to be integrated into a real-time operating system (RTOS) schedule.
• Search Technique: Uses proxylessNAS or hardware-in-the-loop search to obtain ground-truth, worst-case execution time (WCET) measurements on the target silicon, avoiding optimistic simulations. The search space may be constrained to use only fixed-point operations to eliminate unpredictable floating-point overhead.

A hardware cost model is a predictive function—often a lookup table or a small neural network—that estimates the latency, energy, or memory usage of a candidate neural network architecture on specific target hardware. It acts as a fast surrogate for direct hardware measurement during NAS.

• Purpose: Accelerates search by predicting performance without deploying every candidate.
• Types: Can be analytical (based on operation counts) or learned (trained on profiling data).
• Integration: A latency constraint is enforced by using the cost model's prediction as an upper bound in the search objective.

Multi-Objective NAS is an automated design process that optimizes a neural network architecture across multiple competing objectives, such as accuracy, latency, and model size. A latency constraint is a primary objective in this framework.

• Pareto Optimality: The goal is to discover a set of architectures where no single objective can be improved without worsening another.
• Search Strategies: Uses methods like weighted sum, constrained optimization, or evolutionary algorithms to navigate trade-offs.
• Outcome: Produces a frontier of models, allowing developers to select the best architecture for their specific latency-accuracy trade-off.

Hardware-in-the-Loop Search is a NAS methodology where candidate neural network architectures are deployed and profiled on the actual target hardware during the search process to obtain ground-truth performance measurements.

• Accuracy: Provides the most reliable latency measurements, capturing complex hardware interactions.
• Cost: Significantly more time-consuming than using a cost model, as each evaluation requires a full deployment cycle.
• Use Case: Critical for final-stage optimization or when hardware behavior is highly non-linear and difficult to model.

In Neural Architecture Search, the search space defines the set of all possible neural network architectures the algorithm can explore. It is parameterized by operations, layer types, and connectivity patterns.

• Design Impact: A well-designed search space is crucial for enforcing latency constraints; overly large or inefficient spaces make finding fast models difficult.
• Common Elements: Includes choices for operator type (convolution, pooling), kernel size, channel width, and skip connections.
• Pruning: Search space pruning removes inefficient operations to steer the search toward latency-compliant architectures.

In Neural Architecture Search, a performance estimator is a method for predicting the final accuracy of a candidate architecture without fully training it. It works alongside a hardware cost model to evaluate candidates.

• Methods: Includes weight sharing (via a supernet), learning curve extrapolation, and zero-cost proxies.
• Role with Latency: The search algorithm uses the performance estimator for accuracy and the hardware cost model for latency, combining them into a single fitness score.
• Goal: To rapidly identify architectures that are both accurate and meet the latency constraint.

A memory constraint in Hardware-Aware Neural Architecture Search is a limit on the peak RAM or flash memory usage of a candidate model. It ensures the model fits within the strict memory budget of a microcontroller or edge device.

• Co-constraint: Often optimized alongside latency, as the two are frequently correlated (e.g., larger models are slower).
• Measurement: Includes activation memory (runtime RAM) and parameter storage (flash).
• Search Integration: The search algorithm must evaluate both memory footprint and inference time to find a viable model for deployment.