Tensor Arena

The tensor arena's core purpose is to store intermediate activation tensors—the outputs of each neural network layer that become inputs to the next. This avoids the overhead of dynamic heap allocations during inference. For a model with N layers, the arena must be sized to hold the largest intermediate tensor plus any temporaries, not the sum of all tensors, due to in-place computation and memory reuse where possible.

The arena is managed as a linear block of memory. The inference engine's scheduler or memory planner performs a lifetime analysis of all tensors in the model graph. It then allocates slots within the arena, often reusing memory for tensors whose lifetimes do not overlap. This static planning eliminates fragmentation and guarantees deterministic memory usage, which is critical for devices without a Memory Management Unit (MMU).

• Static Allocation: The arena size is fixed at compile-time via a #define (e.g., kTensorArenaSize). This is the most common and reliable method for bare-metal MCUs, ensuring all memory is known at link time and eliminating runtime allocation failures.
• Dynamic Allocation: The arena is allocated from the heap at runtime (e.g., via malloc). This offers flexibility but introduces the risk of allocation failure and must be used cautiously in safety-critical systems.

Arena size is a critical optimization target. It is determined by the peak memory usage of the model, which frameworks like TensorFlow Lite Micro can report. Sizing involves a trade-off:

• Too small: Inference fails.
• Too large: Wastes precious SRAM, preventing other application functions from running. Techniques like operator fusion and quantization directly reduce arena size by shrinking tensor dimensions and data types (e.g., float32 to int8).

The tensor arena is distinct from storage for model parameters (weights and biases). Weights are typically stored in read-only memory (e.g., Flash) as a constant byte array and are streamed into the processor cache as needed. The arena holds only mutable runtime data. Separating weights from activations allows for much larger models, as Flash is usually more abundant than SRAM on an MCU.

Every major TinyML framework uses a tensor arena concept, though the name may differ. It is the central memory pool provided by the developer to the framework's interpreter or scheduler.

• TensorFlow Lite Micro: Passed as tensor_arena to the MicroInterpreter constructor.
• CMSIS-NN: Uses separate buffers for activations, often managed by the user's application code.
• STM32Cube.AI: Automatically generates static activation buffers as C arrays based on the model analysis.

A micro interpreter is the minimal runtime engine within a TinyML framework (e.g., TensorFlow Lite Micro) that manages inference execution. It is responsible for:

• Reading the serialized model structure.
• Planning the execution graph and tensor memory layout.
• Dynamically invoking optimized kernel functions for each layer.
• Managing the tensor arena, allocating and deallocating memory for intermediate activations. Its lightweight design is critical for devices without an operating system.

A FlatBuffer model is a neural network serialized using Google's FlatBuffers library. It is the standard, memory-efficient format for TensorFlow Lite and TensorFlow Lite Micro. Key characteristics include:

• Zero-copy deserialization: Data can be accessed directly from flash memory without loading into RAM, preserving the tensor arena for activations.
• Schema-driven and forward/backward compatible.
• The micro interpreter reads this flat structure to understand the model graph and parameter tensors, which are typically stored in read-only memory (ROM).

Operator fusion is a critical graph optimization that merges consecutive neural network operations (e.g., Conv2D + BatchNorm + ReLU) into a single, compound kernel. This technique directly impacts tensor arena requirements by:

• Eliminating intermediate tensors that would otherwise need allocation between layers.
• Reducing memory bandwidth and kernel invocation overhead.
• It is a key step performed by compilers like TVM or the TFLM converter to minimize peak memory usage during inference.

A C array model is a neural network model represented as a constant C/C++ byte array within a header file, compiled directly into the firmware binary. This approach:

• Eliminates the need for a file system on the microcontroller.
• Stores model weights and architecture in read-only flash memory.
• The inference framework's runtime (e.g., the micro interpreter) references this array in ROM, while the tensor arena in SRAM holds volatile activation data. This separation is fundamental to TinyML memory management.

Graph optimization is the process of transforming a neural network's computational graph to reduce its memory footprint and improve execution speed. For TinyML, this involves:

• Constant folding: Pre-computing static operations.
• Dead code elimination: Removing unused nodes.
• Operator fusion (see related card).
• These optimizations are performed ahead-of-time (AOT) and directly determine the size and lifetime of tensors, thereby defining the minimum required size of the tensor arena.

Memory planning (or tensor allocation) is the compile-time or runtime process of mapping all temporary tensors in a model's execution graph onto a fixed memory block—the tensor arena. Strategies include:

• Greedy by-size algorithms: Allocating the largest tensors first.
• Offline planning: The compiler calculates an optimal static mapping (AOT).
• Lifetime analysis: Overlapping memory for tensors with non-overlapping lifetimes to minimize peak usage. Efficient planning is the difference between a model fitting on-device or not.