Compute Graph

Nodes represent the atomic computational operations (ops) within the neural network. In a microcontroller-optimized graph, these are low-level kernels like:

• Convolution (CONV2D)
• Fully Connected (FULLY_CONNECTED)
• Depthwise Convolution (DEPTHWISE_CONV2D)
• Pooling (AVERAGE_POOL_2D, MAX_POOL_2D)
• Activation functions (RELU, RELU6) Each node is implemented as a highly optimized function (kernel) for the target MCU architecture, often using CMSIS-NN libraries or hand-tuned assembly.

Edges are the directed connections between nodes and represent the multidimensional data arrays (tensors) that flow through the graph. For TinyML, tensor characteristics are critical:

• Data Type: Typically INT8 or INT16 after quantization.
• Shape: Fixed dimensions (e.g., [1, 32, 32, 3] for an image) determined at compile-time.
• Memory Lifetime: Each tensor's allocation is analyzed for static memory planning, allowing buffers to be reused (in-place computation) to minimize peak RAM footprint.

The component that determines the deterministic execution order of all nodes and pre-assigns all tensor memory buffers at compile-time. This eliminates runtime allocation overhead and memory fragmentation. Key features include:

• Operator Fusion: Combines sequences like Conv -> BatchNorm -> Activation into a single fused kernel node.
• Memory Planning: Performs a liveness analysis to overlap tensor lifetimes, allowing a single memory buffer to be reused for multiple, non-concurrent tensors, drastically reducing peak RAM usage.

For integer-only inference, the graph stores the quantization metadata for each tensor edge. This is not the tensor data itself, but the constants needed to convert between integer and real-valued domains:

• Scale (float): The multiplier used in dequantization: real_value = integer_value * scale.
• Zero-Point (int): Used in asymmetric quantization schemes. The integer value that corresponds to the real value zero. These parameters are embedded as constants in the graph, enabling efficient integer-only arithmetic without floating-point hardware.

A mechanism to partition the compute graph for heterogeneous execution. A subgraph is a set of nodes that can be offloaded to a specialized hardware accelerator (e.g., an NPU or DSP). A delegate is the software interface that handles execution of that subgraph. This allows parts of the model (like heavy convolutions) to run on dedicated silicon while the rest executes on the main CPU core, optimizing for both performance and power.

The minimal runtime state required to execute the graph. This is separate from the tensor buffers and includes:

• Pointer to the model's weight array in flash memory.
• Pre-allocated tensor arena: A single, contiguous block of RAM serving as the memory pool for all activation buffers.
• Node execution list: A simple, sequential list of function pointers and buffer indices, generated by the static scheduler, that the lightweight interpreter loops over. This context is typically less than 1KB in size.

Frameworks like TensorFlow Lite Micro (TFLM) and proprietary SDKs construct a compute graph from a trained model file (e.g., .tflite). This involves:

• Parsing the model flatbuffer/protobuf to extract layer definitions.
• Instantiating node objects for each operation (Conv2D, FullyConnected, etc.).
• Creating edges based on tensor producer/consumer relationships.
• Serializing the finalized graph into a format suitable for the microcontroller's memory map, often as a constant C array in flash.

For deterministic execution on MCUs, the graph is statically scheduled at compile-time. This process:

• Determines execution order via a topological sort of the DAG.
• Performs lifetime analysis on all intermediate tensors (activations).
• Allocates memory using a static memory planner that overlays tensors with non-overlapping lifetimes into a single, contiguous arena memory buffer. This minimizes peak RAM footprint and eliminates runtime allocation overhead and fragmentation.

The framework maintains a registry mapping operation types (e.g., kTfLiteConv2D) to optimized kernel functions. During inference:

• The interpreter traverses the scheduled node list.
• For each node, it looks up the appropriate kernel in the registry.
• It dispatches execution to the kernel, passing pointers to the input/output tensors in the arena.
• Kernels are often hand-optimized assembly or CMSIS-NN functions for Arm Cortex-M, leveraging SIMD instructions and fixed-point arithmetic.

Before final code generation, the graph undergoes transformation passes:

• Operator Fusion: Combines sequences like Conv2D -> BatchNorm -> ReLU into a single, fused node to reduce intermediate writes and kernel call overhead.
• Constant Folding: Pre-computes operations on constant tensors (e.g., reshapes) at compile-time.
• Quantization Embedding: Converts floating-point graph nodes into their integer-quantized equivalents (INT8 inference), embedding scaling factors and zero-points as node metadata.
• Redundant Node Elimination: Removes no-op nodes like identity operations.

For systems with heterogeneous cores (e.g., MCU + a tiny NPU), the graph can be partitioned. A compiler analyzes the graph and:

• Labels nodes for execution on specific hardware units (CPU, NPU, DSP).
• Inserts memory transfer operations at partition boundaries to move tensors between memory regions accessible to different cores.
• This requires detailed cost models for communication latency and core capabilities to maximize throughput.

The final implementation integrates tightly with the MCU toolchain:

• The optimized graph and kernels are compiled into a static library.
• The linker script reserves specific sections in flash for the model parameters (.rodata.tensor) and in RAM for the arena (.bss.tensor_arena).
• The application code includes a header file exposing a simple Invoke() API, which calls into the graph interpreter. This creates a standalone, static binary with no external dependencies for deployment.

A compile-time optimization strategy where the execution order of all neural network operations and their memory assignments are determined ahead of time. This eliminates runtime decision overhead, resulting in a completely predictable, low-latency inference loop with deterministic memory usage, which is critical for real-time microcontroller applications.

A compiler optimization that combines multiple sequential neural network layers into a single, fused kernel. Common fusions include:

• Convolution + BatchNorm + Activation
• Fully Connected + Bias + Activation

This reduces intermediate tensor writes to slow external memory, decreases function call overhead, and minimizes the RAM footprint by reusing buffers, leading to significant latency and energy savings.

The manual or automated low-level tuning of fundamental neural network operation implementations for a specific microcontroller architecture. This involves:

• Leveraging SIMD instructions for parallel data processing.
• Applying loop unrolling and loop tiling to improve instruction pipeline efficiency and data cache locality.
• Hand-writing assembly for critical paths. Optimized kernels, such as those in CMSIS-NN, can provide order-of-magnitude speedups over naive implementations.

A memory management strategy where all buffers for model weights, activations, and intermediate tensors are pre-allocated in a fixed, contiguous memory region at compile-time. This eliminates the overhead and potential failure points of dynamic memory allocation (malloc/free), prevents memory fragmentation, and guarantees predictable execution, which is essential for safety-critical embedded systems.