TinyML

The total volatile (RAM) and non-volatile (Flash) memory consumed by the model is the most critical constraint. Microcontroller-class devices often have less than 1 MB of total memory, which must accommodate the model parameters, runtime activations, and the application firmware itself.

• Flash Memory: Stores the model architecture and quantized weights. Limits model size.
• RAM (Working Memory): Holds the model's activations, intermediate tensors, and input/output buffers during inference. Often the more restrictive limit.
• Example: A model for keyword spotting might need to fit within 20 KB of RAM and 100 KB of Flash on an Arm Cortex-M4.

Processing power, measured in clock speed (MHz) and operations per second, restricts model complexity and the feasibility of on-device training. Microcontroller Units (MCUs) lack powerful parallel processors like GPUs.

• Clock Speed: Typically ranges from tens to a few hundred MHz.
• Lack of Parallelism: Most MCUs have single-core CPUs without SIMD extensions, making dense matrix multiplications expensive.
• Impact: Dictates the maximum number of Multiply-Accumulate (MAC) operations per inference, forcing the use of highly efficient neural network architectures like MobileNet or TinyConv.

The total electrical energy available for computation directly defines the operational lifetime of battery-powered devices. Inference and communication are the primary power consumers.

• Milliwatt-Scale Operation: Target power consumption is often in the single-digit milliwatt range for always-on sensing.
• Trade-offs: More complex models or frequent inference increase battery drain. On-device training is an order of magnitude more energy-intensive than inference.
• Goal: Achieve the target task within an energy budget that allows for months or years of battery life, often requiring duty-cycling the sensor and processor.

Many TinyML applications require deterministic, low-latency responses for real-time control or interaction. Inference must complete within a strict deadline.

• Hard Real-Time: Applications like anomaly detection in industrial machinery or wake-word detection require inference in tens of milliseconds.
• Causality: Audio or vibration processing often uses sliding windows, requiring the model to process a window faster than new data arrives.
• Constraint: Latency is a function of compute constraints and model architecture, pushing developers towards smaller, faster models.

The heat generated by computation must be managed within the device's form factor, which often lacks active cooling. Exceeding thermal limits triggers thermal throttling or hardware damage.

• Passive Cooling Only: Small, sealed devices rely on conduction and convection.
• Throttling Impact: To reduce heat, the processor lowers its clock speed, drastically increasing inference time and potentially breaking real-time guarantees.
• Design Implication: Sustained high-compute tasks like on-device training must be carefully scheduled or limited to avoid thermal buildup.

TinyML devices are deployed in uncontrolled, often harsh environments and must operate reliably for years without maintenance. This imposes constraints on software and hardware choices.

• Temperature Extremes: Models and firmware must operate from -40°C to 85°C for industrial applications.
• Memory Integrity: Limited use of dynamic memory allocation (malloc/free) to prevent memory fragmentation over years of uptime.
• Fault Tolerance: The system must recover from power interruptions or sensor noise without human intervention, favoring simple, deterministic software architectures.

A Microcontroller Unit (MCU) is a compact, integrated circuit designed to govern a specific operation in an embedded system. It combines a processor core, memory, and programmable input/output peripherals on a single chip, forming the primary hardware target for TinyML deployments.

• Key Components: Central Processing Unit (CPU), Read-Only Memory (ROM/Flash), Random Access Memory (RAM), and general-purpose I/O pins.
• TinyML Relevance: MCUs provide the ultra-low-power, cost-effective, and physically small compute substrate for always-on sensor intelligence.
• Examples: Arm Cortex-M series (e.g., M0, M4, M7), ESP32, and Arduino boards.

Model compression is a suite of techniques used to reduce the memory and computational requirements of a neural network for deployment on resource-constrained devices. For TinyML, this is not optional but a fundamental design requirement.

Core techniques include:

• Quantization: Reducing the numerical precision of weights and activations (e.g., from 32-bit floats to 8-bit integers).
• Pruning: Removing less important weights or neurons from the model.
• Knowledge Distillation: Training a smaller "student" model to mimic a larger "teacher" model.
• Efficient Architecture Design: Using inherently small models like MobileNetV3 or SqueezeNet.

On-device inference is the process of executing a trained machine learning model locally on an edge device or microcontroller to generate predictions from sensor data, without requiring a network connection to a cloud server.

• Latency & Privacy: Eliminates network round-trip delay and keeps sensitive raw data on the device.
• Energy Trade-off: While communication is often the most power-intensive operation, the compute cost of inference must also fit within the device's energy budget.
• Runtime: Requires a lean inference engine (e.g., TensorFlow Lite for Microcontrollers, CMSIS-NN) compiled into the device's firmware.

An embedded AI accelerator is a specialized co-processor—such as a Neural Processing Unit (NPU), Digital Signal Processor (DSP), or GPU—integrated into a system-on-chip (SoC) to execute neural network operations with far greater efficiency (operations per watt) than a general-purpose CPU.

• Purpose: Offloads intensive matrix multiplications and convolutions from the main MCU core.
• TinyML Impact: Enables more complex models or higher frame rates within strict power envelopes.
• Examples: The Ethos-U55/U65 microNPUs from Arm, the Himax WE-I Plus chip, and DSP cores in the ESP32-S3.

The TinyML software stack is the collection of frameworks, libraries, and tools required to transition a model from development to deployment on a microcontroller. It abstracts hardware complexity for the ML engineer.

Typical layers include:

1. Model Training & Export: PyTorch, TensorFlow, with export to formats like TFLite.
2. Conversion & Optimization: Tools like the TFLite Converter with quantization and pruning.
3. Microcontroller Compilation: The TFLite Micro interpreter, CMSIS-NN libraries, or specialized compilers like Apache TVM.
4. Deployment & Debugging: Integrated Development Environments (IDEs), serial monitors, and logic analyzers.

Sensor fusion is the algorithmic combination of data from multiple physical sensors (e.g., accelerometer, gyroscope, microphone) to produce a more accurate, complete, and reliable understanding of the environment than is possible with a single sensor.

• TinyML Application: A key use case where on-device ML models integrate these fused signals for complex tasks like human activity recognition or anomaly detection.
• Challenge: Requires efficient processing of multiple, continuous sensor data streams in real-time.
• Example: Fusing inertial measurement unit (IMU) data to distinguish between walking, running, and cycling.