Delegation

The core function of a delegate is to identify and offload a contiguous subgraph of operations from the main computational graph. The compiler analyzes the graph, matches patterns of operations that the target hardware can accelerate, and partitions the model. The offloaded subgraph is then executed as a single unit by the delegate's optimized kernel, minimizing transitions between the host runtime and the accelerator.

• Example: A sequence of Conv2D, BatchNorm, and ReLU operations is fused and offloaded to an NPU.
• Benefit: Reduces latency from frequent kernel launches and data marshaling between CPU and accelerator.

Delegation relies on a standardized Hardware Abstraction Layer (HAL) or delegate interface. This interface defines a contract between the main inference runtime (e.g., TFLite Interpreter, ONNX Runtime) and the vendor-provided delegate library. It handles:

• Graph Preparation: Passing the offloaded subgraph and tensor metadata to the delegate.
• Memory Management: Defining whether tensors are allocated by the runtime or the delegate.
• Execution Invocation: Synchronous or asynchronous calls to the delegate's Invoke function.

This abstraction allows the same model to run on different NPUs (e.g., Qualcomm Hexagon, Google Edge TPU) via different delegate binaries without modifying the application code.

Delegates use pattern matching to identify supported operations. The compiler pass scans the computational graph for sequences that map to the accelerator's capabilities. Unsupported operations (e.g., custom layers) remain on the CPU. This creates a partitioned graph where execution jumps between the delegate and the main runtime.

• Critical Constraint: The offloaded subgraph must have well-defined input/output tensors to the CPU graph.
• Challenge: Poor partitioning can lead to excessive memory copies between CPU and accelerator, negating performance gains. Advanced compilers perform cost-based partitioning to minimize these transfers.

Delegation can occur at different stages in the compilation pipeline:

• Ahead-of-Time (AOT) Delegation: The model is fully partitioned, optimized, and compiled for the specific delegate during the build process (e.g., using tflite_convert with a delegate flag). This produces a standalone, efficient binary with minimal runtime overhead.

• Runtime Delegation: The delegate library is linked at runtime. The main runtime (e.g., TFLite Interpreter) loads the model, discovers available delegates, and dynamically partitions the graph. This offers flexibility but adds initialization latency.

• Use Case: AOT is for fixed, production deployments. Runtime delegation is for developer SDKs or devices with multiple optional accelerators.

A key delegation complexity is tensor memory ownership. Strategies include:

• Runtime-Allocated Buffers: The main runtime allocates all input/output tensors. The delegate copies data to/from its internal memory before/after execution. This is simple but adds copy overhead.
• Delegate-Allocated Buffers: The delegate allocates and owns memory for tensors inside the offloaded subgraph. The runtime provides pointers via the abstraction interface. This is more efficient but requires careful alignment and memory type (e.g., shared physical memory) support from the hardware.
• Zero-Copy Buffers: The ideal scenario where the runtime and delegate share the same physical memory (e.g., via a unified memory architecture), eliminating copies entirely.

Robust delegation systems include fallback mechanisms. If a delegate fails to prepare or execute a subgraph (e.g., due to an unsupported dynamic shape), the runtime must gracefully fall back to executing those operations on the CPU. Furthermore, a single model can use multiple delegates for heterogeneous hardware.

• Example: A vision model might delegate convolutional blocks to an NPU, a fully connected layer to a GPU, and a custom post-processing operation to the CPU.
• Orchestration: The compiler and runtime schedule execution across these delegates, managing dependencies and synchronizing results. This is essential for complex System-on-Chip (SoC) architectures common in edge devices.

A delegate is a software interface that allows a compiler runtime to hand off execution of a specific subgraph to an external backend. The compiler identifies supported operation patterns, partitions the graph, and routes subgraphs to the appropriate delegate via a standardized API (e.g., TfLiteDelegate). This abstraction separates model logic from hardware-specific kernel implementations.

• Graph Partitioning: The compiler analyzes the computational graph to find contiguous nodes that can be executed by a delegate.
• Context Switching: The main runtime prepares input tensors, invokes the delegate, and retrieves outputs.
• Fallback Handling: Unsupported operations remain on the CPU or primary backend.

Delegates are implemented to leverage specialized silicon for performance and power efficiency. Common targets include:

• Neural Processing Units (NPUs): Dedicated AI accelerators like the Google Edge TPU or Qualcomm Hexagon DSP.
• GPUs: For parallelizable workloads using APIs like OpenCL or Vulkan.
• Digital Signal Processors (DSPs): For quantized models on mobile SoCs.
• Custom ASICs: Proprietary accelerators in IoT devices.

Each delegate contains highly optimized kernel libraries that exploit the target's unique memory hierarchy and instruction set, often using int8 or fp16 precision for speed.

The compiler performs graph partitioning to decide which operations to delegate. This involves:

• Pattern Matching: Identifying sequences of ops (e.g., CONV2D -> BATCH_NORM -> RELU) that are supported by the delegate's kernel library.
• Dependency Analysis: Ensuring partitioned subgraphs have minimal data transfer across the delegate boundary.
• Cost Modeling: Some compilers use heuristics or profiling data to estimate if delegation reduces latency, as data marshaling overhead can negate benefits for small subgraphs.

Poor partitioning can lead to excessive memory copies between the CPU and accelerator, degrading performance.

Delegation is a foundational feature in edge AI compilers.

• TensorFlow Lite: Uses the TfLiteDelegate C++ API. Examples include the NnApiDelegate for Android NPUs, GpuDelegate for GPUs, and HexagonDelegate for Qualcomm DSPs.
• Android NNAPI: Acts as a system-level delegate, routing subgraphs to available drivers for NPUs, GPUs, and DSPs.
• ONNX Runtime: Employs a similar concept via Execution Providers (EPs) like CUDAExecutionProvider or OpenVINOExecutionProvider.
• TVM: Uses BYOC (Bring Your Own Codegen) to partition graphs for external accelerators.

These frameworks allow vendors to plug in proprietary delegates without modifying the core compiler.

Delegation introduces key engineering trade-offs:

• Latency vs. Overhead: While accelerator execution is fast, data transfer between host and device memory adds overhead. The break-even point depends on subgraph size.
• Memory Footprint: Delegates often require static memory planning on the accelerator, locking dedicated buffers. This can increase the total peak memory usage.
• Power Efficiency: Offloading to a dedicated NPU can be far more power-efficient than running on a general-purpose CPU, crucial for battery-powered edge devices.
• Determinism: Hardware-specific kernels may introduce non-deterministic execution versus standardized CPU math libraries.

Advanced compilers support multiple delegates for a single model, enabling heterogeneous execution across different processors in a system-on-chip (SoC).

• Orchestration: The runtime must manage dependencies and data flow between subgraphs on different backends (e.g., part on NPU, part on GPU).
• Concurrent Execution: Independent subgraphs can be executed in parallel on different accelerators.
• Vendor Extensions: Proprietary compilers (e.g., Qualcomm's SNPE, MediaTek's NeuroPilot) provide advanced multi-delegate scheduling tuned for their specific SoC architectures.

This approach maximizes utilization of all available compute units on complex edge hardware.

A compiler strategy that splits a neural network's computational graph into multiple subgraphs for execution on different processors within a heterogeneous system. This is a higher-level decision that often precedes delegation. The compiler analyzes the graph to decide:

• Which subgraphs are best suited for a dedicated NPU or GPU.
• Which operations should remain on the CPU.
• How data will be transferred between these partitions. Delegation is then used to offload each identified subgraph to its designated hardware backend.

The compiler phase that translates a hardware-agnostic Intermediate Representation (IR) into lower-level IR or machine code specific to a processor or accelerator's capabilities. This is the technical foundation that enables delegation. When a subgraph is delegated to an NPU, the compiler's lowering pass transforms the generic operations into the NPU's native instruction set or a proprietary kernel library call. This process is tightly integrated with the Hardware Abstraction Layer (HAL).

A compilation strategy where a machine learning model is fully optimized, partitioned, and translated into an executable binary for a target device before runtime. AOT compilation is critical for deterministic edge deployment because:

• All delegation decisions (which subgraphs go to which accelerator) are finalized at compile time.
• The resulting binary includes pre-compiled, optimized kernels for each target (CPU, NPU).
• This eliminates Just-In-Time (JIT) overhead, minimizing startup latency and ensuring predictable performance, which is essential for real-time edge applications.

A low-level compiler optimization that combines multiple primitive operations (e.g., Conv2D, BatchNorm, ReLU) into a single, custom-executed kernel. This is a key optimization within a delegated subgraph. By fusing operations:

• Intermediate tensor writes to slow global memory are eliminated.
• Kernel launch overhead is reduced.
• Data locality is dramatically improved. Delegation to an accelerator often enables more aggressive and hardware-specific fusion patterns that wouldn't be possible on a general-purpose CPU.

A compiler optimization that pre-allocates and reuses memory buffers for all tensors at compile time. This is essential for efficient delegation on memory-constrained edge devices. The compiler creates a detailed memory plan that includes:

• Buffer allocations for tensors within each delegated subgraph.
• Memory regions for data transfer between the host CPU and the accelerator.
• This eliminates the latency and fragmentation of dynamic memory allocation during inference, reducing overall memory footprint and improving determinism.