Microcontroller Unit (MCU)

The CPU core is the primary execution unit of the MCU, fetching and executing instructions from memory. For TinyML, the dominant architecture is the ARM Cortex-M series (e.g., Cortex-M0+, M4, M7), a family of 32-bit RISC cores designed for low-power, deterministic operation. Key features include:

• Instruction Set Architecture (ISA): Supports the Thumb/Thumb-2 instruction set for high code density.
• Memory Protection Unit (MPU): Available on some cores (e.g., Cortex-M3+) to enforce memory access rules.
• Digital Signal Processing (DSP) Extensions: Cores like the Cortex-M4 and M7 include SIMD instructions and hardware for multiply-accumulate (MAC) operations, critical for accelerating neural network layers.
• No Operating System Required: Designed to run bare-metal or with a lightweight Real-Time Operating System (RTOS).

MCU memory is split into volatile and non-volatile storage, with severe size constraints measured in kilobytes (KB) to a few megabytes (MB).

• Flash Memory (Non-Volatile): Stores the application code, constant data, and the machine learning model weights. Typical sizes range from 64 KB to 2 MB. Read speeds are a key bottleneck for model loading.
• SRAM (Volatile): Used for runtime data: the model's activations, input/output buffers, and the program's stack/heap. Sizes typically range from 16 KB to 512 KB. The entire inference working set must fit here.
• Tightly Coupled Memory (TCM): Some high-performance MCUs feature ultra-fast SRAM directly connected to the CPU core for critical data and code, minimizing latency.

Peripherals are hardware blocks that interface the CPU with the physical world, enabling sensor data acquisition and control—the essence of edge AI.

• General-Purpose I/O (GPIO): Configurable pins for digital input/output.
• Analog-to-Digital Converters (ADC): Convert real-world analog signals (e.g., from microphones, temperature sensors) into digital values for the ML model.
• Digital Communication Interfaces: I²C, SPI, and UART for connecting to external sensor chips and communication modules.
• Timers/Counters & PWM: Generate precise timing signals and pulse-width modulation for motor control.
• Direct Memory Access (DMA): A controller that transfers data between peripherals and memory without CPU intervention, saving power and cycles for computation.

To overcome the computational limits of the CPU core, modern MCUs increasingly integrate dedicated neural network accelerators.

• Arm Ethos-U55/U65: A micro Neural Processing Unit (microNPU) designed as a coprocessor for Cortex-M systems. It accelerates tensor operations (convolutions, fully connected layers) by orders of magnitude while improving Inferences Per Joule (IPJ).
• Operation: The CPU offloads entire model layers or subgraphs to the accelerator via driver APIs.
• Memory Impact: The accelerator often has its own small, dedicated SRAM (Tile Memory) to hold kernel weights and activation tiles, reducing traffic to main SRAM.
• Toolchain Support: Requires compatible compilers (e.g., TVM, proprietary SDKs) to partition and compile models for the hybrid CPU+NPU architecture.

Power consumption is paramount for battery-operated edge devices. MCUs feature sophisticated power management units (PMUs) and multiple clock domains.

• Multiple Clock Sources: Internal RC oscillators (low power, less accurate) and external crystal oscillators (high accuracy).
• Dynamic Voltage and Frequency Scaling (DVFS): The CPU frequency and core voltage can be scaled down during idle periods or for less intensive tasks to save power.
• Low-Power Modes: Critical for always-on TinyML applications (e.g., keyword spotting).
  • Sleep: CPU halted, peripherals can remain active to wake the system.
  • Deep Sleep: Most of the chip is powered down, with only a few ultra-low-power peripherals (like a GPIO or RTC) active to trigger a full wake-up.
• Wake-Up Time: The latency to transition from a low-power mode to active inference is a key system design parameter.

The on-chip interconnect is the fabric that allows all components (CPU, memory, peripherals, accelerators) to communicate. Its design directly impacts inference latency and determinism.

• AMBA AHB/APB Bus (Arm): A common hierarchical bus architecture. The high-performance AHB connects the CPU, DMA, and main memory. The lower-power APB connects to most peripherals.
• Bus Contention: A potential bottleneck. Simultaneous access attempts from the CPU, DMA, and an accelerator to shared SRAM can stall operations, affecting Worst-Case Execution Time (WCET).
• Memory-Mapped I/O: All peripherals and accelerator control registers are accessed by the CPU as if they were memory locations, using load/store instructions.

MCUs for TinyML typically have kilobytes (KB) of RAM and Flash, not megabytes or gigabytes. This dictates every design decision.

• RAM (Working Memory): Stores the model's activations, intermediate tensors, and input/output buffers during inference. Exceeding RAM causes a hard crash.
• Flash (Storage Memory): Stores the model's weights, the application code, and the inference runtime library. The entire model must fit within available Flash.
• Example: A common target like the ARM Cortex-M4F may have only 256 KB of Flash and 64 KB of RAM, forcing extreme model compression.

MCU clock speeds range from tens to hundreds of MHz, with no parallel cores for general matrix operations. Compute is measured in Millions of Operations Per Second (MOPS).

• Lack of an FPU: Many low-cost MCUs lack a Floating-Point Unit, making 32-bit float operations extremely slow. This necessitates fixed-point arithmetic or 8-bit integer (INT8) quantization.
• DSP Extensions: Some cores (e.g., Cortex-M4, M7, M33) include Digital Signal Processing (DSP) instructions (like SMID) to accelerate multiply-accumulate (MAC) operations, which are fundamental to neural network layers.
• Impact: A single inference on a small vision model can take hundreds of milliseconds, defining the application's real-time capability.

TinyML devices often run on batteries or energy harvesting, operating within a milliwatt (mW) power envelope. Power consumption directly translates to device lifetime.

• Active Current: The current draw during inference, which spikes with CPU/DSP activity. Optimizing for lower inference latency often reduces active time, saving power.
• Sleep Current: The minimal current draw when the MCU is in deep sleep, which can be in microamps (µA). The system design aims to maximize sleep time.
• Key Metric: Inferences Per Joule (IPJ) measures the total computational work achieved per unit of energy, the ultimate efficiency score for battery-powered AI.

Embedded systems for industrial control or safety require predictable, bounded execution times. Non-deterministic behavior is unacceptable.

• Worst-Case Execution Time (WCET): The maximum possible time an inference task can take must be known and guaranteed. This precludes the use of dynamic memory allocation (malloc/free) which can cause fragmentation and variable timing.
• Static Memory Allocation: All model tensor buffers are allocated at compile-time within a contiguous memory arena, ensuring no runtime allocation failures and predictable memory layout.
• Real-Time Operating Systems (RTOS): Often used to schedule inference tasks alongside other critical system functions with strict priority levels.

Many TinyML deployments use bare-metal programming or a minimal RTOS, not a full OS like Linux. This removes overhead but places responsibility on the developer.

• No Virtual Memory: There is no memory paging or swapping. The entire model and runtime must fit in physical memory.
• Direct Hardware Access: The inference engine must manage peripherals (ADCs, I2C for sensors) directly or via simple drivers.
• Toolchain Constraints: Development uses specialized, lightweight toolchains like Arm GCC or Arm Compiler 6, with debugging often done via JTAG/SWD probes.

Getting sensor data into the MCU and results out can be a primary bottleneck, often overshadowing compute time.

• Sensor Sampling: Reading data from an accelerometer or microphone via I2C, SPI, or I2S consumes CPU cycles and power. Efficient drivers and Direct Memory Access (DMA) are critical.
• Output Actuation: Triggering a relay, sending a message via UART, or blinking an LED is the end-goal. The inference latency directly impacts the system's responsiveness.
• System-on-Chip (SoC) Integration: Modern MCUs for AI (e.g., with an Arm Ethos-U55 microNPU) integrate the accelerator, sensor interfaces, and radio (BLE) on one chip to minimize these bottlenecks.

Tiny Machine Learning (TinyML) is the subfield of machine learning focused on developing and deploying models that can perform inference directly on microcontrollers and deeply embedded devices. These models operate under extreme constraints, typically requiring:

• Memory footprints under 100KB
• Power consumption in the milliwatt range
• Latency measured in milliseconds TinyML enables intelligence in always-on, battery-powered applications like predictive maintenance sensors, keyword spotting, and visual wake words, where cloud connectivity is impractical.

The ARM Cortex-M series is the dominant family of 32-bit RISC processor cores designed for microcontroller applications. They are the primary CPU architecture for TinyML due to their:

• Ultra-low power consumption (often microamps in sleep mode)
• Deterministic execution for real-time systems
• Extensive software ecosystem, including CMSIS Common series include the high-performance Cortex-M7, balanced Cortex-M4 (often with DSP extensions), and ultra-efficient Cortex-M0+. Their ISA and memory hierarchy directly constrain model design choices.

In TinyML, memory footprint refers to the total amount of RAM (working memory) and Flash/ROM (storage memory) a machine learning model and its runtime require. This is the primary constraint for MCU deployment.

• Flash Memory: Stores the model's weights, architecture, and constant data. Typically ranges from 32KB to 512KB on target MCUs.
• RAM (Activation Memory): Holds the model's input, output, and intermediate tensors during inference. Often limited to 32KB-256KB. Optimization techniques like quantization and pruning directly target reducing this footprint to fit within an MCU's fixed memory map.

Milliwatt computing is the paradigm of designing software to operate within a power budget of milliwatts (thousandths of a watt). This is essential for TinyML devices that are:

• Battery-powered and must last for months or years
• Energy-harvesting, powered by sources like solar, vibration, or RF Key strategies include:
• Extreme duty cycling, where the MCU sleeps >99% of the time
• Optimizing inferences per joule (IPJ), the metric of computational work per unit energy
• Leveraging low-power accelerators like the Arm Ethos-U55 microNPU to complete inference faster, allowing the core to return to sleep sooner.

Inference latency is the time delay between presenting an input to a model on an MCU and receiving the output prediction. For real-time TinyML applications, this must be predictable and fast.

• Typical Targets: Audio keyword spotting (<20ms), visual wake words (<200ms).
• Determinism: MCU runtimes often use static memory allocation to avoid garbage collection delays, ensuring a consistent Worst-Case Execution Time (WCET). Latency is reduced through operator fusion, fixed-point arithmetic, and hardware acceleration (e.g., using DSP blocks for convolutions).