uTensor

uTensor directly imports models trained in TensorFlow or Keras, converting them into a memory-efficient format for microcontrollers. The framework parses the standard SavedModel or Keras .h5 format, extracting the computational graph and weights.

• Conversion Process: Uses a Python converter script to transform the model into C++ source files.
• FlatBuffer Support: Internally uses a lightweight serialization similar to FlatBuffers to store model architecture and parameters without external dependencies.
• Graph Translation: Maps common TensorFlow operations (like Conv2D, DepthwiseConv2D, FullyConnected, ReLU) to their uTensor kernel equivalents.

The core of uTensor is a header-only C++ library designed for extreme portability and minimal footprint. It provides a simple API to load and run models without dynamic memory allocation (heap usage).

• Static Memory Planning: Allocates a contiguous block of memory (a tensor arena) at compile-time for intermediate activations.
• Simple API: Core usage involves just a few calls: model = uTensor::load_model() and model->invoke().
• Zero OS Dependencies: Runs on bare-metal systems or with any real-time operating system (RTOS), requiring only a standard C++ compiler (C++11 or later).

uTensor includes a library of hand-optimized kernel functions for common neural network operations, written in efficient C/C++ and often using fixed-point arithmetic.

• Fixed-Point Quantization: Kernels primarily operate on 8-bit or 16-bit integer data types to avoid the overhead of floating-point units (FPUs) on low-cost MCUs.
• Hardware-Specific Optimizations: While portable, kernels can be extended or replaced with assembly-optimized versions for specific architectures (e.g., Arm Cortex-M with DSP extensions).
• Common Ops Supported: Includes optimized implementations for convolutions, pooling, fully connected layers, and activation functions like ReLU and softmax.

The framework is engineered to operate within kilobytes of RAM, using several strategies to minimize memory overhead during inference.

• Tensor Arena: A single, statically-sized memory buffer holds all intermediate tensors. The runtime performs in-place operations and reuses memory aggressively.
• Lazy Tensor Allocation: Tensors are only allocated in the arena immediately before they are needed as an operation's input.
• Graph-Level Optimization: Applies operator fusion (e.g., fusing a convolution with a subsequent ReLU activation) to reduce the number of intermediate tensors created.

uTensor is designed to be highly portable across a wide range of 32-bit microcontroller architectures and development toolchains.

• Processor Support: Primarily targets Arm Cortex-M series (M0, M3, M4, M7) but can be ported to other cores like RISC-V or ESP32.
• Build System Integration: Integrates easily with common embedded build systems like Arm Mbed, PlatformIO, Zephyr RTOS, and Makefile-based projects.
• Vendor Independence: Does not require proprietary tools or SDKs, making it suitable for open-source and commercial projects across multiple silicon vendors.

The deployment workflow is streamlined, converting a trained model directly into compilable C++ code that becomes part of the firmware binary.

• Two-Phase Conversion: 1) A Python script converts the .pb or .h5 model into C++ header/source files. 2) These files are added to the MCU project.
• C Array Model Output: The model weights and architecture are stored as constant C arrays within the code, eliminating the need for a file system on the device.
• End-to-End Example: The open-source repository provides complete examples for tasks like keyword spotting and image classification, demonstrating the full path from training to on-device inference.