Heterogeneous Compute

Heterogeneous compute partitions a single AI workload across the CPU for sequential control logic, the GPU for parallel throughput, and the NPU for sustained, energy-efficient tensor operations.

• A medical imaging pipeline might use the NPU for real-time organ segmentation, the GPU for 3D volume rendering, and the CPU for DICOM protocol handling.
• This avoids the thermal throttling and battery drain that would occur if a single processor handled the entire workload.

The system's runtime scheduler analyzes a neural network's computational graph and assigns each operator to the optimal processor based on a cost model.

• A convolution layer with high data reuse is mapped to the NPU's systolic array.
• A non-supported custom activation function falls back to the CPU.
• The scheduler minimizes data movement overhead, which often dominates energy consumption more than the computation itself.

Efficient heterogeneous compute relies on a unified memory architecture where the CPU, GPU, and NPU share a single physical address space.

• This eliminates the need to explicitly copy tensors between separate memory pools, a process that can add milliseconds of latency.
• The hardware employs cache coherency protocols to ensure that a weight matrix updated by the CPU is immediately visible to the NPU without a cache flush.

A dynamic power management controller monitors the thermal envelope and allocates a power budget to each processor in real-time.

• For a battery-operated wearable, the system may cap the GPU frequency and offload sustained inference to the more efficient NPU to meet a strict latency budget without exceeding thermal limits.
• This enables continuous health monitoring without the device becoming uncomfortably warm for the patient.

A model is not directly portable across heterogeneous hardware. A compiler stack transforms a high-level graph into optimized, processor-specific machine code.

• Operator fusion combines a convolution, batch normalization, and ReLU activation into a single NPU kernel to eliminate intermediate memory reads and writes.
• The compiler applies Int8 quantization specifically for the NPU's integer pipeline while keeping precision-sensitive layers on the GPU.

Heterogeneous compute enables inter-processor pipelining, where different stages of a streaming inference task execute concurrently on different hardware.

• In a digital stethoscope, the NPU continuously processes an audio spectrogram for anomaly detection while the CPU simultaneously encrypts and transmits the previous result.
• This overlapping execution hides the latency of each individual stage, maximizing throughput for real-time diagnostic streams.

A specialized hardware accelerator designed from the transistor level up to execute the multiply-accumulate (MAC) operations central to neural network inference. Unlike general-purpose CPUs, NPUs leverage systolic array architectures and massive parallelism to achieve tera-operations-per-second throughput at milliwatt power budgets, making them essential for continuous AI on battery-operated medical wearables.

A compression technique that reduces the numerical precision of a model's weights and activations from 32-bit floating-point (FP32) to 8-bit integers (INT8) or lower. This directly enables a model to be dispatched to an NPU's integer-optimized pipelines, reducing memory bandwidth pressure and eliminating the need for expensive floating-point units. The process requires a calibration dataset to determine optimal clipping ranges that minimize accuracy degradation.

A graph-level optimization that combines multiple discrete neural network layers—such as a convolution, batch normalization, and ReLU activation—into a single compound kernel. By eliminating intermediate memory reads and writes between operations, fusion drastically reduces the memory bandwidth bottleneck on edge SoCs. This is a critical compiler pass that enables a model to fully saturate the parallel compute units of a GPU or NPU.

An advanced silicon design that performs matrix-vector multiplication directly inside the memory array using analog computation, rather than shuttling data between separate memory and compute units. By eliminating the von Neumann bottleneck, this architecture achieves an order-of-magnitude improvement in energy efficiency for AI inference. It is particularly suited for always-on, ultra-low-power medical sensors performing continuous health monitoring.

A design paradigm where the model architecture itself is automatically optimized not just for accuracy, but for the specific latency, memory, and power constraints of the target heterogeneous SoC. The search algorithm incorporates a hardware cost model that penalizes operations unsupported by the NPU or that cause excessive CPU fallback. This co-design approach yields models that are natively efficient on the target chip from the first training run.