Wake-on-Inference

The trigger model is a minimal, purpose-built machine learning model that runs on the always-on coprocessor. It is the decision engine that determines when to wake the main system. Key characteristics include:

• Extreme compression: Heavily quantized (often to 8-bit or lower), pruned, and distilled to fit within the coprocessor's tight SRAM constraints (often < 100KB).
• Binary or low-class output: Designed not for detailed classification, but to detect a specific event (e.g., "wake word present," "anomalous vibration detected") with high precision to avoid false wakes.
• Optimized for low Joule per Inference: Every operation is counted. The model architecture (e.g., depthwise separable convolutions) and kernels are co-designed with the hardware to minimize energy. A successful detection generates an interrupt, triggering the wake-up sequence for the main AI accelerator.

The main AI accelerator is the high-performance, power-hungry processor (e.g., a GPU, large NPU, or DSP cluster) that performs the complex, primary inference task. In a wake-on-inference system, it resides in a separate power domain that is aggressively managed:

• Power-gated when idle: The accelerator's power rail is completely switched off by the PMU, eliminating both dynamic power and static power (leakage power).
• Rapid wake-up: Upon a trigger, the accelerator must power on, load the full application model from memory, and begin inference with minimal latency. This transition is a critical design parameter.
• Performance burst: It executes the primary task (e.g., natural language query, image classification) at full speed, often using Dynamic Voltage and Frequency Scaling (DVFS) to finish quickly and return to a deep sleep state.

This component handles the raw data flow that feeds the trigger model. Its design is critical for both functionality and efficiency:

• Low-power sensor front-end: Interfaces with analog or digital sensors (IMUs, PDM microphones) using minimal energy.
• Smart sampling & buffering: Often uses duty cycling to sample sensors intermittently, storing data in a circular buffer. The always-on coprocessor processes data from this buffer.
• Event-driven activation: In some architectures, the sensor interface itself can be configured with simple thresholds (e.g., accelerometer magnitude), acting as a preliminary hardware filter to wake the coprocessor only when interesting data is present, saving further energy. This subsystem ensures the trigger model has access to the necessary context without requiring the entire data pipeline to be active continuously.

Memory architecture is pivotal in minimizing energy during the wake-up sequence. Key considerations include:

• Non-volatile memory (Flash): Stores the compressed trigger model and the larger application model for the main accelerator. Access energy is high, so models are loaded strategically.
• Fast, low-power SRAM on the coprocessor: Holds the active trigger model and its data buffers. Sized precisely to avoid wasteful over-provisioning.
• Main system memory (DRAM): Often placed in a self-refresh sleep state when idle. Waking it contributes significantly to the energy trace of a wake event. Optimized on-device model formats help reduce the volume of data that must be transferred. The goal is to keep the energy-intensive main memory in a low-power state as long as possible, fetching only essential data for the imminent inference.

The primary consumer application. A dedicated, ultra-low-power Digital Signal Processor (DSP) or microcontroller runs a tiny keyword spotting model (e.g., "Hey Siri," "Okay Google") continuously on microphone input. Only upon a high-confidence detection is the main Application Processor and its large automatic speech recognition model powered up for full query processing. This reduces the always-listening power draw from watts to milliwatts.

Enables 24/7 monitoring without prohibitive energy costs. A low-power vision processing unit analyzes a low-frame-rate, low-resolution video stream. It runs a lightweight motion detection or person detection model. The main AI accelerator and high-resolution camera sensor are activated only when a person or vehicle is classified, triggering recording or an alert. This extends battery life for wireless cameras from days to months.

• Key Benefit: Eliminates false alarms from non-relevant motion (e.g., trees, animals).
• Hardware: Often uses microNPUs like the Hailo-8 or Ethos-U55 for the always-on task.

Deployed on wireless vibration or acoustic sensors attached to machinery. A microcontroller with a TinyML model continuously analyzes sensor data for anomalous patterns indicative of impending failure (e.g., bearing wear). The device remains in a deep sleep state, transmitting only summary health status. The full sensor suite and high-bandwidth radio (e.g., Wi-Fi) wake only to stream detailed diagnostic data when an anomaly threshold is crossed, conserving battery and network bandwidth.

• Model Type: Anomaly detection or regression models predicting remaining useful life.
• Protocol: Often uses low-power wide-area networks (LPWAN) like LoRaWAN for status pings.

Enables continuous health tracking. A sensor hub microcontroller runs simple models on photoplethysmogram (PPG) or accelerometer data to detect events like atrial fibrillation, falls, or the onset of sleep. The main processor and higher-accuracy models (e.g., for heart rate variability analysis) are triggered only by these events. This architecture is fundamental to achieving week-long battery life in smartwatches and medical patches.

• Example: Detecting a sudden fall via accelerometer pattern, then waking the main CPU to initiate an emergency call with GPS location.
• Challenge: Requires highly optimized models to fit in the SRAM of the always-on domain.

Creates responsive, energy-autonomous environments. A battery-powered door/window sensor uses a wake-on-inference acoustic model to distinguish between normal ambient noise and the specific sound of glass breaking, triggering a full alarm. An environmental sensor might run a simple model to detect patterns indicative of a water leak (from humidity/temperature) before waking to send an alert.

• Power Source: Often designed for energy harvesting (solar, kinetic) or multi-year coin cell battery life.
• Stack: Built on frameworks like TensorFlow Lite for Microcontrollers.

Enhances safety without constant high-power compute. A low-power image signal processor or dedicated core runs a basic face detection or eye presence model on data from an interior camera. The full driver attention monitoring system—with complex gaze estimation and drowsiness detection models—is activated only when a driver is confirmed to be present and the vehicle is in motion. This reduces system heat and energy draw.

• Integration: Part of the vehicle's domain controller architecture.
• Standard: Aligns with ISO 26262 functional safety requirements for partitioned systems.

Always-on sensing refers to the continuous, low-power operation of sensors and a minimal inference pipeline (e.g., for keyword spotting or simple motion detection) to enable devices to perceive and react to events without requiring user interaction or waking the main processor. This subsystem is the sensory front-end for wake-on-inference architectures.

• Key Components: Ultra-low-power microcontrollers (MCUs), microNPUs, and analog front-ends.
• Example: A smart speaker's microphone array and keyword-spotting model that constantly listens for "Hey Google" or "Alexa" while consuming only milliwatts.

Event-driven inference is an execution paradigm where model inference is triggered only by specific, predefined external events, rather than running continuously. This is the software execution model that wake-on-inference implements.

• Triggers: Sensor data exceeding a threshold, a timer expiration, or a simple inference result from an always-on subsystem.
• Contrast with Polling: Eliminates the energy waste of periodic, scheduled inference runs, waiting instead for a genuine signal.
• System Impact: Requires careful state management and low-latency wake-up mechanisms to transition from sleep to active compute states.

Power gating is a circuit-level technique that completely shuts off power to inactive or idle blocks of a silicon chip, eliminating both dynamic and static (leakage) power consumption in those regions. It is the most aggressive form of power saving.

• Application in Wake-on-Inference: The main AI accelerator (GPU, NPU) and high-performance CPU cores are fully power-gated when the always-on subsystem is monitoring.
• Implementation: Uses header or footer switches (MOSFETs) to disconnect a power domain from the supply rail (Vdd).
• Trade-off: Introduces a latency and energy penalty to restore power and re-initialize the block, which must be amortized over the subsequent computation.

Duty cycling is a power management strategy where a system periodically alternates between short active periods (for computation or sensing) and long sleep periods. It reduces average power consumption by limiting active time.

• Contrast with Wake-on-Inference: Duty cycling is time-based and periodic, while wake-on-inference is event-based and asynchronous.
• Hybrid Use: Often used within the always-on subsystem itself (e.g., sampling a sensor at 10 Hz instead of continuously).
• Key Parameter: The duty cycle percentage, calculated as (Active Time / Total Period Time) * 100%.

Joule per inference is a direct energy efficiency metric that measures the total energy, in joules, required to perform a single forward pass (inference) of a machine learning model on a given hardware platform. It is the fundamental unit for evaluating wake-on-inference savings.

• Calculation: Energy (Joules) = Average Power (Watts) * Inference Latency (Seconds).
• System-Level View: For wake-on-inference, the total system energy for a task includes the energy of the always-on monitor plus the energy of the main accelerator's inference.
• Design Goal: Minimize this metric through model compression, efficient hardware, and architectural techniques like wake-on-inference.

A Power Management Unit (PMU) is a dedicated hardware block or integrated circuit responsible for generating, regulating, sequencing, and controlling the supply voltages and power states of various components within an electronic system. It is the orchestrator of wake-on-inference.

• Core Functions:
  • Voltage Regulation: Provides stable, often variable, supply voltages (Vdd) to different domains.
  • Power Sequencing: Controls the order in which domains power up/down to prevent latch-up or data corruption.
  • State Control: Manages transitions between sleep, idle, and active states based on requests from the always-on processor.
• Integration: In modern SoCs, the PMU is often integrated alongside the always-on microcontroller domain.