Heterogeneous Compute

A heterogeneous system integrates multiple instruction set architectures (ISAs) and specialized cores on a single system-on-chip (SoC). Each processor type is optimized for a specific computational pattern:

• CPU: Handles control flow, sequential logic, and orchestration tasks with low latency
• GPU: Executes massively parallel matrix multiplications for dense neural network layers
• NPU: Accelerates convolution and transformer operations with systolic array architectures at minimal power
• DSP: Processes signal transforms and audio/vision pre-processing with fixed-point arithmetic This diversity eliminates the bottleneck of forcing all workloads through a single general-purpose core.

The primary metric for edge inference is not raw throughput but inferences per joule. Heterogeneous compute achieves this by mapping each computational kernel to its most efficient hardware:

• Data reuse: NPUs exploit weight stationary dataflows to minimize DRAM access energy
• Precision matching: INT8 operations on NPUs consume 10-30x less energy than FP32 on GPUs
• Duty cycling: Idle accelerators are power-gated independently, unlike monolithic GPU architectures
• Memory locality: Tightly coupled scratchpad memories on DSPs avoid cache coherency overhead This fine-grained resource allocation enables sustained inference on battery-powered devices.

Unlike homogeneous SIMD parallelism, heterogeneous systems exploit multiple concurrent execution streams across different processor types:

• Pipeline parallelism: A video pipeline processes ISP demosaicing on DSP, object detection on NPU, and tracking logic on CPU simultaneously
• Model parallelism: A large transformer splits attention heads across multiple NPU cores while the CPU manages key-value cache
• Workload interleaving: While the GPU runs a batch matmul, the DSP concurrently executes audio keyword spotting This task-level concurrency maximizes silicon utilization and reduces end-to-end latency for multi-modal applications.

Modern heterogeneous SoCs employ a physically unified, logically shared memory subsystem to eliminate explicit data copies between processors:

• Zero-copy buffers: NPU and CPU share pointer-based access to the same tensor in DDR
• Hardware coherency: Bus protocols like AMBA ACE maintain cache coherency across CPU clusters and accelerators
• IOMMU/SMMU: System memory management units provide virtual-to-physical address translation for all processing elements This architecture avoids the PCIe bottleneck of discrete GPU setups, reducing data movement latency from milliseconds to microseconds.

Heterogeneous execution requires a unified compiler framework that partitions a computation graph across available hardware at compile time:

• Operator placement: The compiler assigns each op to the optimal processor based on a cost model of latency and energy
• Graph partitioning: Subgraphs are extracted for each accelerator, with automatic insertion of data transfer nodes
• Fusion rules: Consecutive element-wise ops are fused into single kernels to minimize intermediate tensor materialization
• Auto-tuning: Black-box optimization searches the space of tiling factors and loop orders for each kernel Frameworks like TVM, MLIR, and IREE provide the multi-target code generation necessary for this complexity.

Safety-critical edge applications demand deterministic latency guarantees that heterogeneous architectures can provide through hardware isolation:

• Bandwidth reservation: Memory controllers allocate guaranteed DRAM bandwidth to the NPU for inference while the CPU handles sporadic control tasks
• Interrupt affinity: Time-critical sensor interrupts are routed directly to the DSP without OS scheduler involvement
• Preemption: Lower-priority GPU rendering tasks are preempted at tile boundaries when a high-priority inference request arrives
• Worst-case execution time (WCET) analysis: Static analysis tools bound the maximum latency of each accelerator kernel These mechanisms enable heterogeneous systems to meet hard real-time deadlines in automotive and industrial applications.