TinyML Runtime

The runtime pre-allocates all memory buffers during model initialization, eliminating dynamic heap allocation at inference time. This prevents memory fragmentation and guarantees deterministic execution on bare-metal systems without an operating system. The memory plan is computed offline by the interpreter, mapping tensors to fixed SRAM addresses. This allows the runtime to operate within a strict SRAM footprint of less than 100KB, avoiding stack overflows that would crash the microcontroller.

The runtime executes models as a flat directed acyclic graph of operators rather than relying on dynamic Python execution. Each neural network layer is registered as a kernel function that operates on input and output tensors. The interpreter iterates through this graph sequentially, calling each operator's implementation. This architecture eliminates the overhead of a full framework runtime and allows for operator fusion, where adjacent operations like convolution and activation are merged into a single kernel call to reduce memory round-trips.

A thin HAL (Hardware Abstraction Layer) decouples the runtime core from the specific microcontroller architecture. This layer provides standardized interfaces for:

• Clock cycles and timing for profiling
• UART or serial debugging for logging
• Hardware-specific optimized kernels via CMSIS-NN or vendor libraries The HAL enables the same model to run across Arm Cortex-M, RISC-V, and proprietary DSP cores without modifying the application logic, only swapping the backend kernel implementations.

The runtime includes hand-optimized assembly or intrinsic-based implementations for INT8 and lower precision arithmetic. These kernels exploit SIMD (Single Instruction Multiple Data) instructions to perform multiple multiply-accumulate operations per clock cycle. For example, CMSIS-NN provides drop-in kernels for depthwise convolutions and fully connected layers that achieve up to 4.6x performance improvement over naive C implementations on Cortex-M4 processors. The kernel library must support per-channel and per-tensor quantization schemes.

The runtime ingests models serialized in the FlatBuffers binary format, which enables zero-copy deserialization directly from flash memory. Unlike Protocol Buffers or JSON, FlatBuffers do not require a parsing step—the model graph, weights, and metadata are accessed via pointer arithmetic on the raw byte buffer. This eliminates the RAM overhead of constructing an intermediate representation. The format supports schema evolution, allowing newer runtime versions to load older models without recompilation.

To minimize binary size, the runtime uses an operator resolver pattern where only the kernels explicitly registered by the developer are linked into the final firmware. The resolver maps op codes from the model file to concrete function pointers at initialization. This allows the firmware to exclude unused operations—if a model does not use LSTM or softmax, those kernels are stripped by the linker. The resulting binary can be as small as 16KB of flash for a minimal keyword spotting application.