CMSIS-NN

CMSIS-NN kernels are hand-optimized in assembly and C to exploit the specific instruction sets of Arm Cortex-M cores (M0, M3, M4, M7, M33, M55). This includes:

• Use of SIMD (Single Instruction, Multiple Data) instructions on supported cores (e.g., Armv7E-M) for parallel data processing.
• Efficient utilization of the processor's pipeline and register file to minimize stalls.
• Loop unrolling and software pipelining techniques to reduce instruction overhead. The result is near-theoretical peak performance for fundamental operations like convolution and matrix multiplication on the target CPU.

CMSIS-NN is designed for 8-bit and 16-bit fixed-point (integer) arithmetic, which is essential for microcontrollers lacking Floating-Point Units (FPUs) or where FPU usage is too power-intensive.

• Kernels use Q7 (int8) and Q15 (int16) data formats.
• Implements saturation arithmetic to prevent overflow and underflow.
• Provides scaling functions to manage the fixed-point representation of weights and activations. This integer-only approach drastically reduces memory footprint, increases computational speed, and lowers power consumption compared to floating-point inference.

The library is architected to operate within the severe SRAM constraints (often < 512KB) of Cortex-M devices.

• In-place operations where possible to reuse memory buffers for activations.
• Partial computation strategies that process large tensors in tiles to fit within a small, fast local memory footprint.
• Static memory allocation patterns that allow developers to precisely size and place buffers, avoiding heap fragmentation.
• Minimized stack usage through careful register allocation and function design. This focus enables the execution of meaningful neural networks without external memory.

CMSIS-NN provides a set of discrete, optimized functions for core neural network operations, allowing developers to build custom inference pipelines.

• Convolution kernels: arm_convolve_HWC_q7_fast() for depthwise and regular convolutions.
• Fully connected layers: arm_fully_connected_q7().
• Activation functions: arm_relu_q7(), arm_nn_activations_direct_q15().
• Pooling: arm_max_pool_s8(), arm_avgpool_s8().
• Basic math: arm_nn_mat_mult(), arm_softmax_q7(). This modularity allows integration into various inference runtimes (like TFLM) or use as standalone building blocks in custom C firmware.

For systems featuring an Arm Ethos-U55 or Ethos-U65 microNPU, CMSIS-NN provides a unified software interface.

• The CMSIS-NN API can delegate appropriate operations (e.g., convolutions) to the hardware accelerator.
• Maintains a single software pipeline that can target the CPU, NPU, or a hybrid of both.
• Allows for performance scaling on more capable system-on-chips (SoCs) without a complete application rewrite. This feature future-proofs code, enabling it to leverage dedicated AI accelerators as they become available in microcontroller-class devices.

Many silicon vendors building microcontrollers around Arm Cortex-M cores integrate CMSIS-NN into their proprietary SDKs to provide optimized AI capabilities. For example:

• Espressif's ESP-DL library for ESP32-S3 can leverage CMSIS-NN-style optimizations.
• NXP's eIQ® ML software for i.MX RT crossover MCUs often includes CMSIS-NN as a backend option.
• Infineon's ModusToolbox™ ML middleware uses CMSIS-NN for targets like the PSoC™ 6 MCU. These SDKs wrap CMSIS-NN with higher-level APIs, board support packages, and driver integrations specific to their hardware.