Model Partitioning

The primary goal is to map different parts of a computational graph to the most suitable processor in a system. For example:

• Convolutional layers are often offloaded to an NPU or GPU for parallel compute.
• Control flow operations (e.g., conditionals, loops) may remain on the CPU.
• Lightweight pre/post-processing (e.g., image resizing, non-max suppression) can be assigned to a DSP or CPU. This maximizes overall system efficiency by matching operator semantics to hardware strengths.

Partitioning can occur at different levels of abstraction:

• Operator-level: Each individual neural network operation (e.g., Conv2D, MatMul) is assigned to a device. This offers fine-grained control but high communication overhead.
• Subgraph-level: Groups of consecutive operators (a fused kernel or logical block) are assigned as a unit. This is the most common approach, balancing efficiency with manageable communication.
• Layer-level: Entire neural network layers are assigned. This is coarser and simpler but may miss optimization opportunities within a layer. The compiler's graph optimizer typically identifies optimal subgraph boundaries.

The partitioning decision is driven by a cost model that estimates:

• Compute Latency: Execution time for a subgraph on each available processor.
• Memory Transfer Cost: Time to move tensor data between devices (e.g., over a PCIe bus or shared memory).
• Power Consumption: Energy cost of execution on different accelerators. Heuristics or solvers (e.g., dynamic programming, greedy algorithms) use this model to find a partition that minimizes total end-to-end latency or a combined objective like latency-per-watt.

A critical constraint is the cost of transferring data between partitions on different devices. The compiler must:

• Minimize Cross-Device Edges: Place frequently communicating operators on the same device to avoid expensive data transfers.
• Insert Synchronization: Ensure data dependencies are respected across device boundaries, which may require explicit synchronization primitives.
• Optimize Tensor Layouts: Convert tensor formats (e.g., NHWC to NCHW) during transfer if required by the target accelerator's kernel libraries. Poor partitioning can result in the system being communication-bound, where transfer time dominates compute time.

Static Partitioning is determined at compile time (AOT). The model binary contains pre-defined subgraphs for each device. This minimizes runtime overhead and is used for fixed hardware targets. Dynamic Partitioning can adapt at runtime (JIT). This is useful for:

• Systems where hardware availability changes (e.g., a cooling GPU).
• Varying input shapes or batch sizes that change the optimal partition.
• Adaptive systems that profile actual execution times and re-partition for subsequent inferences. Dynamic partitioning adds runtime decision logic but can improve robustness in variable environments.

Model partitioning is a compiler optimization that divides a single computational graph into multiple, distinct subgraphs. The primary goal is to map each subgraph to the most suitable processor in a heterogeneous system (e.g., CPU, GPU, NPU, DSP). This maximizes overall system efficiency by executing each operation on the hardware best suited for it, balancing latency, throughput, and power consumption. It is essential for edge devices where resources are constrained and workloads are diverse.

Compilers use specific criteria and algorithms to decide where to split the graph:

• Operator Support: Checks if a hardware backend has a kernel implementation for a given operator.
• Performance Cost Model: Estimates the execution latency or power cost of running a subgraph on each available processor.
• Data Transfer Overhead: Models the cost of moving tensor data between different memory spaces (e.g., from CPU to NPU RAM).

Common algorithms include greedy algorithms that make local optimal choices and dynamic programming approaches that search for a globally optimal partition plan.

Partitioning is implemented via a delegate mechanism. The main runtime (e.g., TFLite Interpreter, ONNX Runtime) identifies a subgraph marked for a specific accelerator and delegates its execution. The delegate is a hardware-specific library (like the NNAPI Delegate for Android NPUs or the GPU Delegate for mobile GPUs) that takes the subgraph, optimizes it further, and manages execution. This creates a clear abstraction between the high-level model and low-level hardware drivers.

• Maximized Hardware Utilization: Leverages all available silicon, offloading compute-intensive ops (e.g., convolutions) to NPUs while keeping control flow on CPUs.
• Reduced End-to-End Latency: Parallel execution across processors can significantly cut inference time.
• Improved Energy Efficiency: Executing ops on specialized, lower-power accelerators extends battery life in mobile and IoT devices.
• System Responsiveness: Keeps the main CPU free for other application tasks by offloading AI workloads.

• Partitioning Overhead: The time spent analyzing the graph and making partitioning decisions must be less than the performance gain.
• Memory Transfers: Excessive data shuttling between processors can become a bottleneck, negating acceleration benefits. Compilers must co-locate dependent operations.
• Kernel Availability: Partitioning is only possible if efficient kernels exist for the target hardware.
• Dynamic Shapes: Models with input-dependent shapes complicate static partition planning, often requiring fallback paths or JIT recompilation.

TensorFlow Lite uses a Delegates API (e.g., XNNPACKDelegate for CPUs, GpuDelegate). The compiler (tflite_convert) can annotate the model, but final partitioning often occurs at runtime.

ONNX Runtime uses its Execution Provider interface. A model can be partitioned across multiple providers (e.g., CPUExecutionProvider and TensorrtExecutionProvider) using a graph analyzer to find supported subgraphs.

Apache TVM performs sophisticated graph partitioning as part of its relay module, using a cost-based model to split the graph before performing target-specific lowering and code generation.

A compiler mechanism where a subgraph of operations is offloaded from the main runtime to be executed by a dedicated hardware accelerator or a highly optimized kernel library via a delegate interface. It is the primary execution mechanism for partitioned subgraphs.

• Key Difference: Partitioning identifies which subgraphs can run where; delegation implements the offload to the specific hardware backend.
• Example: A TFLite model uses the NNAPI or GPU delegate to run specific operator patterns on an Android device's Neural Processing Unit or graphics processor.

A compiler pass that transforms a neural network's computational graph by applying high-level transformations to improve execution efficiency. It is a prerequisite step that often makes partitioning more effective.

• Common Optimizations: Operator fusion, constant folding, and dead code elimination.
• Synergy with Partitioning: Fusing a Conv2D, BatchNorm, and ReLU into a single compound operator creates a more efficient, atomic unit that can be cleanly partitioned onto an accelerator.

A software layer within a compiler stack that provides a standardized interface for generating code and managing resources across diverse hardware accelerators. It is the critical interface that enables partitioned subgraphs to execute on different processors.

• Function: Abstracts vendor-specific driver APIs (e.g., for an NPU, GPU, or DSP) into a common set of functions for memory management and kernel execution.
• Example: The TensorFlow Lite runtime uses HALs to allow the same model delegate to work across different chipset vendors' NPUs.

The process of compiling a software program on one computer platform (the host) to run on a different platform with a distinct architecture (the target). It is the broader build process that incorporates partitioning.

• Role in Edge AI: Developers partition and compile models on powerful servers (x86) to produce executables for ARM-based edge devices or microcontrollers.
• Toolchain Integration: Compilers like TVM and MLIR perform partitioning and other optimizations during the cross-compilation workflow.

A compiler optimization that pre-allocates and reuses memory buffers for all tensors at compile time. It is essential for partitioned models to minimize overhead when data moves between processors.

• Benefit: Eliminates dynamic memory allocation overhead, crucial for deterministic performance on resource-constrained edge devices.
• Challenge in Partitioning: The planner must account for memory buffers that act as intermediate tensors between subgraphs running on separate processors (e.g., CPU->NPU).

A compilation strategy where a machine learning model is fully optimized and translated into an executable binary for a target device before runtime. It is the preferred deployment method for partitioned edge models.

• Advantages: Minimizes startup latency, allows for aggressive whole-program optimizations (including optimal partitioning), and reduces the runtime binary size.
• Contrast with JIT: Just-In-Time compilation performs optimization at runtime, which is often too costly for edge devices with limited compute.