NNAPI Delegation

MediaTek's NeuroPilot SDK and its AI Processing Units (APUs) are designed with NNAPI delegation as a core deployment target. Their system-on-chips (SoCs) provide:

• Dedicated NPU Cores: Hardware designed for efficient matrix and convolution operations.
• NeuroPilot Delegate: A vendor-specific delegate that often sits under the NNAPI delegate, providing further optimizations for MediaTek silicon.
• Multi-Core Heterogeneity: APUs can split workloads across different types of cores (e.g., performance vs. efficiency) managed via NNAPI preferences.

Qualcomm's Snapdragon platforms feature the Hexagon Tensor Processor (HTP), a dedicated NPU. NNAPI delegation routes supported ops to the HTP via the Qualcomm driver stack.

• Hexagon NN Direct: While Qualcomm offers its proprietary Snapdragon Neural Processing Engine (SNPE) SDK, NNAPI provides a standardized alternative.
• Power Efficiency: Delegation to the Hexagon NPU significantly reduces power consumption compared to CPU/GPU execution for sustained AI tasks.
• Quantization Optimization: The HTP is highly optimized for INT8 and mixed-precision inference, aligning with NNAPI's quantization support.

Google's custom Tensor SoC (used in Pixel phones) integrates the Edge TPU as a primary AI accelerator. NNAPI delegation is the standard mechanism for accessing it from Android apps.

• Pixel-Only Optimization: The tight hardware/software co-design allows for highly efficient driver communication and memory management via NNAPI.
• Google's TensorFlow Lite Runtime: The default ML stack on Pixel devices is optimized to use NNAPI delegation to the Edge TPU seamlessly. This represents a vertically integrated example of NNAPI's role in connecting frameworks to custom silicon.

Samsung Exynos SoCs incorporate a dedicated NPU which is exposed to the Android runtime via NNAPI. Key aspects include:

• Tri-Cluster NPU Design: Some Exynos NPUs use a multi-core architecture for balancing throughput and efficiency, managed through NNAPI's abstraction.
• Samsung ONE SDK: While offering a full proprietary SDK, NNAPI delegation provides a vendor-agnostic path for framework-integrated models.
• Device-Specific Tuning: NNAPI allows Samsung to optimize driver performance for its own hardware without requiring app developers to write custom code.

The NNAPI Hardware Abstraction Layer (HAL) is a vendor-implemented interface that sits between the Android NNAPI runtime and the actual accelerator driver (e.g., for an NPU or DSP). It is the critical bridge for delegation:

• Vendor Responsibility: Chip manufacturers (Qualcomm, MediaTek, Google) implement the HAL to expose their hardware's capabilities.
• Operation Mapping: The HAL declares which neural network operations (ops) and data types (e.g., INT8, FP16) it supports for acceleration.
• Driver Isolation: It abstracts low-level driver details, allowing NNAPI to work with diverse hardware through a standardized interface.

The Android Neural Networks API is the foundational Android C/C++ API that defines the standard for on-device machine learning operations. NNAPI delegation is a feature of its runtime:

• Runtime & Driver: Consists of the NNAPI runtime (in Android framework) and the vendor HAL/driver. The runtime manages delegation decisions.
• Operation Set: Defines a comprehensive list of neural network operations (e.g., CONV_2D, FULLY_CONNECTED, LSTM) that hardware can support.
• Execution Model: Supports asynchronous, burst, and caching modes. Delegation occurs when the runtime partitions a model graph, sending subgraphs to available accelerators via the HAL.

TFLite Delegates are the pluggable components that enable TensorFlow Lite to offload computational work to accelerators. The NNAPI Delegate is one specific implementation:

• Delegate Pattern: The TFLite interpreter analyzes the model graph and passes supported subgraphs to a registered delegate for execution.
• Types of Delegates:
  • NNAPI Delegate: For Android hardware acceleration.
  • GPU Delegate: For cross-platform GPU acceleration (OpenGL/Vulkan).
  • Hexagon Delegate: For Qualcomm Hexagon DSPs (often superseded by NNAPI).
  • Custom Delegates: For proprietary accelerators or experimental hardware.
• Fallback Mechanism: Operations not supported by the delegate fall back to the CPU for execution.

Model Quantization is a compression technique that reduces the numerical precision of a model's weights and activations. It is intrinsically linked to NNAPI delegation because most mobile accelerators are optimized for low-precision math:

• Hardware Requirement: NPUs and DSPs commonly have superior performance and power efficiency for INT8 or FP16 operations versus FP32.
• Delegation Prerequisite: A model must often be quantized (e.g., to INT8 via Post-Training Quantization or Quantization-Aware Training) for the NNAPI HAL to accept and accelerate it.
• Performance Trade-off: Enables faster inference and lower power consumption at the potential cost of a minor accuracy drop, which is managed through calibration.