Model Serving

The core runtime that executes the trained model on input data. For TinyML, this is a highly optimized library (e.g., TensorFlow Lite Micro, CMSIS-NN) that performs fixed-point or integer arithmetic, manages the model's layers in memory, and interfaces with the microcontroller's hardware. It must handle:

• Memory-mapped model execution to avoid loading the entire model into RAM.
• Kernel optimizations for the target CPU or NPU (e.g., ARM Cortex-M DSP instructions).
• Deterministic latency guarantees required for real-time sensor applications.

A centralized system for storing, versioning, and serving model artifacts (.tflite, .onnx files). In a microcontroller context, this includes:

• Compressed artifact storage for quantized and pruned models.
• Metadata tracking (e.g., input/output schema, accuracy metrics, target hardware).
• Dependency management for the specific inference engine version and required system libraries.
• A/B testing support by serving multiple model variants to different device cohorts.

The interface through which the device receives input and returns predictions. For constrained devices, this is not a traditional REST API but often a lightweight protocol:

• MQTT or CoAP for publish-subscribe messaging in IoT networks.
• Custom binary protocols over UART, SPI, or BLE for direct sensor communication.
• Request batching at the gateway level to aggregate inferences from multiple devices.
• The API layer must handle serialization/deserialization of sensor data (e.g., int16 arrays from an accelerometer) into the model's expected tensor format.

Handles the dynamic settings and operational state of the model on the device. This component is critical for updates and telemetry.

• Manages runtime parameters (e.g., inference frequency, sensor sampling rate).
• Stores the active model version and canary deployment flags.
• Implements Desired State Configuration, where a central server defines the target state (model version, config), and the device agent reconciles its local state to match.
• Often integrated with a Real-Time Operating System (RTOS) task scheduler.

A lightweight process that collects and reports metrics from the device for observability. Key metrics include:

• Inference latency and peak memory usage per prediction.
• System health (CPU load, battery voltage, free heap).
• Data drift indicators (e.g., statistical properties of input sensor values).
• Prediction logs or confidence scores for shadow mode deployments.
• Data is sent intermittently via the communication layer to a central monitoring dashboard.

Manages the secure installation of new models and firmware. For TinyML, this involves:

• Over-the-Air (OTA) update mechanisms using differential updates to minimize bandwidth.
• Secure boot and digital signature verification of the new model artifact before installation.
• Atomic updates with rollback capability to prevent bricking devices.
• Canary deployment logic, gradually rolling out a new model to a subset of the fleet while monitoring for performance regressions or crashes.

Microcontrollers (MCUs) typically have kilobytes (KB) of RAM and megahertz (MHz) clock speeds, orders of magnitude less than cloud servers. This imposes strict limits on:

• Model Size: Models must be compressed via quantization and pruning to fit into flash storage.
• Runtime Memory: The activation memory (tensor buffers) during inference must fit entirely within volatile RAM, prohibiting large batch sizes or complex intermediate representations.
• Compute Budget: Limited CPU cycles demand highly optimized kernels using fixed-point arithmetic and hand-tuned assembly for common operations like convolutions.

Many TinyML devices are battery-powered or energy-harvesting, making power the primary constraint. Serving must be ultra-efficient.

• Inference Cost: Energy per prediction is a critical Key Performance Indicator (KPI). Techniques like model sparsity and selective execution are used to minimize active compute time.
• Sleep States: The system must spend most of its time in deep sleep modes. The serving runtime must wake the device, perform inference, and return to sleep with minimal overhead.
• Dynamic Voltage and Frequency Scaling (DVFS): The serving system may need to manage the MCU's clock speed and voltage in real-time to balance latency and energy use.

TinyML devices often operate in environments with unreliable or absent network connectivity. The serving architecture must be inherently offline-first.

• On-Device Inference: All model serving is local; the device cannot fall back to a cloud API. The model must be fully self-contained.
• Deferred Communication: Telemetry (e.g., prediction logs, health metrics) must be queued locally and transmitted in bursts when a connection is available, using protocols like MQTT.
• OTA Update Complexity: Over-the-Air updates for new models must be robust against connection drops, verified via digital signatures, and minimize downtime and energy use during the update process.

A deployed model may need to run across a heterogeneous fleet of MCUs from different vendors (e.g., Arm Cortex-M, RISC-V, ESP32) with varying capabilities.

• Compiler Toolchains: Model serving code must be portable across different C/C++ compilers (GCC, Arm Clang, IAR).
• Hardware Acceleration: Leveraging Neural Processing Unit (NPU) coprocessors or Digital Signal Processors (DSPs) requires vendor-specific kernels and a Hardware Abstraction Layer (HAL).
• Performance Consistency: Ensuring consistent latency and accuracy across different hardware targets requires extensive benchmarking and profiling.

Unlike data centers, edge devices face unpredictable physical conditions.

• Sensor Noise and Failure: The serving system must handle corrupt or missing input data gracefully, potentially requiring pre-processing sanity checks or fallback logic.
• Temperature and Voltage Variation: MCU performance can drift with temperature, affecting inference timing. Servicing systems may need temperature-aware calibration.
• Physical Security: Devices are vulnerable to tampering. Serving runtimes may need to integrate with a Trusted Execution Environment (TEE) or use secure boot to verify model integrity before execution.

Managing the deployment, monitoring, and updating of models across thousands to millions of devices is a logistical challenge.

• Fleet Health Monitoring: Collecting performance metrics and detecting model drift or data drift in a power-efficient manner.
• Phased Rollouts: Using canary deployments to slowly roll out new models to a subset of devices, monitoring for regressions before full fleet deployment.
• Configuration Drift: Ensuring all devices maintain the correct model version and configuration settings over years of operation, often managed via a desired state configuration system.

A method for wirelessly distributing new firmware, software, or machine learning models to a remote fleet of microcontroller devices. For TinyML, this is critical for deploying model updates without physical access.

• Key Components: A secure update server, a compressed model artifact (e.g., a TensorFlow Lite for Microcontrollers file), and a client bootloader on the device.
• Challenges: Must operate over low-bandwidth networks (e.g., LPWAN), manage limited device storage for dual images, and ensure update integrity and atomicity to prevent bricking.
• Example: Updating an anomaly detection model on 10,000 smart sensors in a factory after retraining on new failure patterns.

A centralized repository for storing, versioning, and managing the lifecycle of machine learning model artifacts. In a TinyML pipeline, it tracks which quantized model version is deployed to which device cohort.

• Functions: Stores metadata like training dataset hash, accuracy metrics, memory footprint, and target hardware architecture (e.g., ARM Cortex-M4).
• Integration: Links to CI/CD pipelines to automatically promote a model from staging to production after passing embedded benchmarks.
• Critical for: Auditing, rollback capabilities, and ensuring consistency across a heterogeneous device fleet.

An operating system designed for deterministic, time-critical applications on embedded systems and microcontrollers. It provides the foundational scheduling and resource management for reliable model inference.

• Key Features: Preemptive multitasking, predictable task scheduling, and a small memory footprint (often under 10KB).
• Role in TinyML: Manages concurrent tasks like sensor data sampling, inference execution, and network communication, guaranteeing the model meets its real-time latency SLO.
• Examples: FreeRTOS, Zephyr, and Azure RTOS ThreadX are common choices for TinyML deployments.

A software layer that provides a uniform interface for application code to interact with hardware components. It is essential for making TinyML models portable across different microcontroller families.

• Purpose: Abstracts hardware-specific details like memory-mapped I/O registers, DSP instructions, and neural accelerator APIs (e.g., Arm CMSIS-NN).
• Benefit: Allows the same quantized model code to run on an ESP32, an nRF52840, or an STM32 with minimal porting effort.
• Implementation: Often provided by the microcontroller vendor or as part of an RTOS, defining standard interfaces for GPIO, I2C, SPI, and timers.

The process of systematically handling changes to a system's settings and parameters using declarative files and version control. For microcontroller fleets, this governs model versions, inference parameters, and system settings.

• Manifests: A device's configuration is defined in a file specifying the model version, sensor sampling rates, and telemetry reporting intervals.
• Tools: Systems like Terraform or purpose-built fleet managers use these manifests to enforce a desired state configuration across thousands of devices via OTA updates.
• Ensures: Consistency, reproducibility, and the ability to roll back both application logic and model parameters to a known-good state.

A lightweight, publish-subscribe network protocol designed for efficient machine-to-machine communication in constrained environments. It is the de facto standard for telemetry and command/control in TinyML deployments.

• Advantages: Minimal packet overhead, works reliably over high-latency or low-bandwidth cellular/LPWAN networks, and supports offline-first operation with retained messages.
• Use Case: Devices publish inference results (e.g., sensor/device123/anomaly_score) to a broker and subscribe to topics to receive new model binaries or configuration updates.
• Ecosystem: Often paired with a broker like Mosquitto or a cloud service (AWS IoT Core, Azure IoT Hub) for scalable fleet management.