Arm NN

Arm NN provides a unified Hardware Abstraction Layer (HAL) that decouples neural network frameworks from the underlying hardware. This allows a single model, defined in a framework like TensorFlow or PyTorch, to run efficiently across heterogeneous Arm compute units without modification.

• Backend Delegation: The runtime automatically delegates subgraphs or operations to the optimal backend (e.g., CPU, GPU, NPU) based on operator support and performance profiles.
• Vendor Neutrality: Provides a consistent API for application developers while silicon partners can plug in optimized backend implementations for their specific IP.

The engine acts as a universal translator, supporting models from multiple major training frameworks via parsers and converters.

• Native Parsers: Includes built-in parsers for TensorFlow, TensorFlow Lite, ONNX, and Caffe. This eliminates the need for developers to manually convert models into a proprietary intermediate representation.
• Arm NN Converter: A standalone tool that converts models from supported frameworks into Arm NN's optimized internal graph format (.armnn). This conversion applies initial optimizations like constant folding and scheduler hinting.

Arm NN performs a series of graph-level optimizations to transform a naive model graph into one highly optimized for Arm hardware.

• Layer Fusion: Combines consecutive operations (e.g., Convolution + BatchNorm + Activation) into a single, compute-efficient kernel. This reduces memory bandwidth and kernel launch overhead.
• Constant Folding: Pre-computes parts of the graph that rely only on constant values, turning them into static tensors.
• Scheduler Optimization: Reorders and schedules operations to maximize data locality and minimize memory transfers between different compute units.

A flagship feature is the deep, optimized integration with the Arm Ethos-N series of Neural Processing Units (NPUs). Arm NN includes a dedicated backend that compiles and executes subgraphs on the Ethos-N for extreme efficiency.

• Command Stream Generation: Translates neural network operations into highly efficient, proprietary command streams executed directly by the NPU.
• Weight Encoding: Compresses and packs model weights into formats (like WeightStream) that are optimal for the NPU's memory system and compute architecture.
• Supported Operations: Optimizes for Ethos-N's strengths, including INT8 and INT16 quantized convolutions, depthwise convolutions, and fully connected layers.

For devices featuring Arm Mali GPUs, Arm NN provides a backend leveraging the Arm Compute Library (ACL). This delivers high-performance inference for floating-point and quantized models.

• OpenCL Backend: Utilizes the GPU via OpenCL, with kernels tuned for Mali's Midgard and Bifrost architectures.
• Optimized Kernels: The Compute Library provides hand-optimized OpenCL kernels for common neural network operators, ensuring efficient use of the GPU's parallel compute units and memory hierarchy.

Even for CPU-only inference, Arm NN provides significant performance gains over naive framework runtimes on Arm Cortex-A processors.

• Neon Intrinsics: Heavily utilizes Arm Neon SIMD (Single Instruction, Multiple Data) instructions via the Arm Compute Library to accelerate linear algebra operations.
• Multi-threading: Implements efficient multi-threading strategies to leverage all available CPU cores for parallelizable layers.
• Cache-aware Algorithms: Kernels are designed to be cache-friendly, minimizing costly accesses to main memory, which is critical for power-efficient mobile and edge devices.