Micro Interpreter

This component is responsible for reading the serialized neural network model from storage (e.g., a FlatBuffer or C array in ROM) and parsing it into an in-memory representation of the computational graph. It must perform this with zero dynamic memory allocations and minimal code footprint. The parser validates the model schema and extracts metadata such as tensor shapes, operator types, and the execution order.

The single most critical resource manager. The tensor arena is a statically allocated, contiguous block of SRAM (often 10s of KB) that acts as a scratchpad for all intermediate activation tensors during inference. The interpreter pre-plans a memory mapping to overlay tensors with non-overlapping lifetimes, a technique called in-place or static memory planning, to minimize peak RAM usage. Efficient arena management is the difference between a model fitting on-device or not.

A lightweight lookup table that maps each neural network operator type (e.g., CONV_2D, DEPTHWISE_CONV_2D, FULLY_CONNECTED) to its corresponding optimized kernel function. The dispatcher invokes these kernels in the sequence defined by the model's graph. Kernels are often hand-optimized assembly or intrinsic functions (like those in CMSIS-NN) for maximum performance. This registry is typically compiled in, avoiding dynamic linking overhead.

The component that traverses the parsed model's computational graph and executes nodes in the correct order. In a micro interpreter, scheduling is typically static and determined at compile-time or model-load time. It handles data dependencies between operators and ensures tensors are ready before a kernel is dispatched. For complex models, it may apply graph optimizations like operator fusion (combining a convolution, batch norm, and activation into one kernel) at this stage to reduce overhead.

The collection of highly optimized, low-level functions that perform the actual mathematical computations. These kernels are the performance heart of the interpreter and are tailored for:

• Fixed-point arithmetic (int8, int16) instead of floating-point.
• Specific microcontroller CPU architectures (Arm Cortex-M, RISC-V).
• Leveraging CPU-specific SIMD instructions (e.g., Arm Helium, DSP extensions).
• Hardware accelerators like a microNPU (e.g., Arm Ethos-U55) if present. Kernel quality directly defines the system's latency and energy efficiency.

A thin, C-language application programming interface that provides the only entry points for the embedded application. Core functions typically include:

• InterpreterInit(): Sets up the tensor arena and loads the model.
• InterpreterInvoke(): Triggers a single inference pass.
• GetInputTensor() / GetOutputTensor(): Provides pointers to input/output buffers. This API is designed for deterministic real-time behavior, with no hidden allocations, threading, or system calls, making it safe for bare-metal and RTOS environments.

A code-generation-based interpreter that produces specialized, in-place kernels. Its strategy is:

• Kernel Specialization: Generates C code where loops are unrolled and tensor dimensions are hard-coded for the specific deployed model.
• In-Place Computation: Reuses memory buffers across layers to drastically reduce peak memory usage, a technique critical for devices with < 512KB SRAM.
• Patch-based Inference: For vision models, it processes input images in small patches to keep activation memory within SRAM limits, avoiding external RAM.

All micro interpreters share core constraints that shape their architecture:

• No Dynamic Allocation: All memory (tensor arena, kernel contexts) must be statically or stack-allocated.
• Minimal C++/RTTI: Often written in C or a C++ subset to avoid heavy standard library overhead.
• Single-Threaded Execution: Assumes a bare-metal, non-preemptive environment.
• Deterministic Latency: Must avoid non-deterministic operations (e.g., cache misses are managed, no OS paging).
• Fault Tolerance: Often includes bounds checking and null pointer guards, as a crash means a device reboot.

Operator fusion is a critical graph optimization often applied before a model reaches the micro interpreter. It combines consecutive neural network layers (e.g., a Convolution, BatchNorm, and Activation) into a single, compound kernel. For a micro interpreter, this optimization:

• Reduces Kernel Invocations: Lowers the interpreter's dispatching overhead.
• Minimizes Intermediate Tensor Storage: Fused operations pass data directly, shrinking the required tensor arena.
• Enables Hardware-Specific Optimizations: Allows hand-optimized assembly for common layer sequences.

The tensor arena is a contiguous block of memory (typically SRAM) that the micro interpreter manages for activation tensors—the intermediate data between network layers. Its size is a major constraint in TinyML deployment.

• Static vs. Dynamic Allocation: Can be pre-allocated (deterministic) or dynamically managed by the interpreter.
• Memory Planning: The interpreter performs graph scheduling to overlay tensors in memory where possible, minimizing peak usage.
• Performance Bottleneck: Insufficient arena size will cause inference to fail, making its optimization paramount.

A micro-compiler (e.g., TVM's MicroTVM, vendor NPU compilers) is a tool that performs ahead-of-time (AOT) compilation. It transforms a neural network model into highly optimized, low-level code before deployment, which the micro interpreter then executes. This contrasts with a pure interpreter that decodes operations at runtime.

• Reduces Runtime Overhead: Moves graph planning and optimization to compile-time.
• Generates Lean Code: Outputs minimal C code or machine code for the target MCU.
• Enables Advanced Optimizations: Can unroll loops, inline functions, and apply hardware-specific instructions.