Hardware Delegate

The delegate's primary function is to identify and offload a connected set of operations (a subgraph) from the main CPU execution path to a specialized hardware accelerator. This is determined via graph partitioning during the graph lowering phase. The compiler analyzes the computational graph, identifies operations with compatible kernels on the target hardware (e.g., convolutions on an NPU), and partitions the graph accordingly. The delegate then manages data transfer and execution for this isolated subgraph.

Delegates provide a standardized interface between the high-level inference framework (e.g., TensorFlow Lite, PyTorch) and vendor-specific hardware drivers. This abstraction allows model developers to write hardware-agnostic code. The framework's operator dispatch logic calls the delegate's unified API, which then translates the request into proprietary commands for the accelerator (e.g., OpenCL for GPUs, vendor SDK for NPUs). This separates model logic from hardware-specific implementation details.

A delegate contains or links to optimized kernels (pre-compiled computational routines) for the target hardware. It performs kernel auto-tuning or selection at runtime based on:

• Input tensor shapes (inferred via shape inference)
• Data types (e.g., FP16, INT8)
• Available hardware resources For example, a GPU delegate might select different CUDA kernel configurations for large vs. small batch sizes. This ensures the most efficient kernel variant is used for the given context.

The delegate handles the complex memory orchestration between the host (CPU/RAM) and the accelerator. This includes:

• Static memory planning for tensors within the delegated subgraph to minimize allocations.
• Scheduling asynchronous memory transfers (DMA) to overlap computation with data movement.
• Managing pinned memory buffers for efficient host-to-device transfers.
• Potentially applying data layout optimization (e.g., NHWC to NCHW) to match the accelerator's preferred memory format, reducing transpose overhead.

Delegates often implement fallback mechanisms for operations unsupported by the hardware. If a critical operation within a candidate subgraph cannot be executed on the accelerator, the delegate may:

• Split the subgraph, offloading only the compatible portion.
• Execute the entire subgraph on the CPU as a fallback. This enables hybrid execution, where parts of a model run on an NPU (e.g., vision layers) and other parts run on the CPU (e.g., custom post-processing), ensuring functional correctness even with partial hardware support.

Effective delegates are tightly integrated with the framework's compiler optimization passes. Prior to delegation, the graph undergoes transformations like operator fusion, constant folding, and dead code elimination to create larger, more efficient subgraphs for offloading. For example, a sequence of Conv2D, BatchNorm, and ReLU may be fused into a single operation that maps perfectly to a dedicated NPU kernel. The delegate works on this optimized graph, maximizing the computational workload handed to the accelerator.

The algorithmic process of dividing a large computational graph into smaller, executable subgraphs. This is the precursor step for a hardware delegate, which identifies the clusters of operations suitable for offload to a specific accelerator.

• Key Function: Analyzes operator dependencies and data flow to create valid, efficient partitions.
• Delegation Boundary: The partition boundary becomes the delegation point where the host runtime hands off execution to the delegate.
• Objective: Minimizes data transfer between the host CPU and the accelerator while maximizing the compute workload on the specialized hardware.

The runtime mechanism responsible for selecting and invoking the correct implementation (kernel) for an operation. A hardware delegate acts as a specialized dispatcher for its target accelerator.

• Dispatch Table: The runtime maintains a registry of available delegates (e.g., GPU, NPU, DSP).
• Selection Logic: Based on the operator type, data formats, and availability, the runtime routes execution to the appropriate delegate.
• Fallback Handling: If a delegate cannot execute an operation, the framework must have a fallback path (often to the CPU) to ensure functional correctness.

The process of transforming a high-level, hardware-agnostic intermediate representation (IR) of a model into a lower-level, accelerator-specific IR or machine code. A delegate performs its own internal lowering for its target hardware.

• High-Level IR: Framework-defined graph (e.g., TensorFlow GraphDef, PyTorch FX Graph).
• Delegate IR: The delegate converts its assigned subgraph into a proprietary format optimized for its hardware (e.g., TensorRT plans, Core ML models, Qualcomm SNPE containers).
• Optimization Passes: Lowering often applies hardware-specific optimizations like kernel fusion and data layout transformations.

A compilation strategy where the model's computational graph is fully optimized and compiled to a target-specific binary before deployment. Hardware delegates are often AOT-compiled for maximum performance.

• Performance vs. Flexibility: AOT compilation eliminates runtime compilation overhead, yielding predictable, fast inference but less flexibility for dynamic input shapes.
• Delegate Artifact: The output is a pre-compiled binary or library (the delegate) that the main inference runtime loads and executes.
• Use Case: Standard for mobile and edge deployment where startup latency and binary size are critical constraints.

An automated process that searches for the optimal implementation parameters for computational kernels on specific hardware. Delegates for complex accelerators like GPUs heavily rely on auto-tuning.

• Parameters Tuned: Tile sizes, work-group dimensions, loop unrolling factors, and memory access patterns.
• Deployment Phase: Tuning can occur during AOT compilation or on first run (caching the results).
• Goal: Maximizes hardware utilization (compute throughput, memory bandwidth) for the delegate's kernels, which directly impacts inference speed and power efficiency.

A compile-time optimization that pre-allocates and reuses memory buffers for tensors within a computational graph. An efficient delegate performs static memory planning for its assigned subgraph.

• Lifetime Analysis: The delegate analyzes the tensor lifetimes within its partition to identify overlapping memory regions that can be reused.
• Peak Memory Reduction: Minimizes the peak working memory required on the accelerator, which is often a scarce resource.
• Zero-Copy Goal: Advanced planning aims to enable zero-copy data transfer between fused operations within the delegate, a key performance benefit.