Sensor Fusion

The non-linear extension of the Kalman Filter. It linearizes the system's non-linear dynamics and measurement models around the current state estimate using a first-order Taylor series expansion.

Core Limitation: The linearization can introduce significant errors for highly non-linear systems, leading to divergence. It is widely used for sensor fusion in robotics (e.g., fusing wheel odometry with LiDAR) and remains a standard for moderately non-linear problems where computational resources are constrained.

A more robust alternative to the EKF for non-linear systems. Instead of linearizing, the UKF uses a deterministic sampling technique (the Unscented Transform).

• Selects a minimal set of sigma points around the mean.
• Propagates these points through the true non-linear functions.
• Computes a new mean and covariance from the transformed points.

Advantage: It captures the posterior mean and covariance accurately to the 3rd order for any non-linearity, versus the EKF's 1st order. This makes it superior for systems with strong non-linearities, such as orientation estimation using quaternions.

A simple, frequency-domain fusion method that is exceptionally efficient for microcontroller deployment. It combines two sensors with complementary noise characteristics:

• A high-pass filter is applied to a high-frequency, drift-prone sensor (e.g., gyroscope for angular rate).
• A low-pass filter is applied to a low-frequency, stable sensor (e.g., accelerometer for gravity vector).

The outputs are summed. For example, in attitude estimation: Angle ≈ α * (Gyro_Integrated_Angle) + (1-α) * (Accel_Measured_Angle). The cutoff frequency (α) determines the blend. Its low computational cost makes it ideal for TinyML applications on basic MCUs.

A sequential Monte Carlo method ideal for highly non-linear and non-Gaussian estimation problems. It represents the state estimate using a set of random samples (particles), each with an associated weight.

• Prediction: Particles are propagated through the non-linear system model.
• Update: Weights are updated based on how well each particle's predicted measurement matches the actual sensor reading.
• Resampling: Low-weight particles are replaced by duplicates of high-weight particles.

Key Use Case: Simultaneous Localization and Mapping (SLAM) in robotics, where the environment and robot pose are both unknown and the state space is complex and multi-modal.

These define the high-level data flow and hierarchy for combining multiple sensors:

• Centralized Fusion: All raw sensor data is sent to a single fusion node. Provides optimal performance but requires high bandwidth and central processing.
• Decentralized/Distributed Fusion: Each sensor node performs local processing and sends partial estimates to a fusion center. More robust to node failure, reduces communication load.
• Hierarchical Fusion: A hybrid approach with multiple fusion layers (e.g., low-level fusion of IMU sensors, then high-level fusion with camera data).

TinyML Consideration: Decentralized architectures align with edge computing paradigms, pushing preprocessing to the sensor node to minimize data transmission and central MCU load.

Self-driving cars rely on a sensor suite—cameras, LiDAR, radar, and IMUs—to create a coherent 3D model of the environment. Sensor fusion algorithms (e.g., Kalman filters, particle filters) are used to:

• Localize the vehicle precisely within a map using GPS, IMU, and wheel odometry.
• Track dynamic objects (pedestrians, other vehicles) by fusing camera-based classification with radar-based velocity and range data.
• Provide redundancy; if one sensor fails (e.g., camera blinded by sun), others maintain situational awareness.

Mobile robots and UAVs use sensor fusion for state estimation and environmental interaction. A typical system fuses:

• IMU Data (accelerometer, gyroscope) for high-frequency attitude and acceleration.
• Visual Odometry from cameras for drift-corrected position.
• Ultrasonic/LiDAR for precise obstacle avoidance and altitude hold.
• Magnetometer for global heading reference (correcting gyro drift). This creates a stable inertial navigation system (INS) that allows for precise hovering, autonomous flight in GPS-denied environments, and delicate manipulation tasks.

Smartwatches and fitness bands use low-power sensor fusion to infer user activity and health metrics from miniature MEMS sensors.

• Activity Recognition: Fusing 3-axis accelerometer and gyroscope data to classify walking, running, cycling, and sleep stages.
• Heart Rate Monitoring: Combining optical PPG (photoplethysmography) sensor data with accelerometer data to filter out motion artifacts, providing accurate heart rate readings during exercise.
• Fall Detection: Analyzing sudden changes in acceleration and orientation to detect potential falls, especially for elderly users.

AR/VR headsets require ultra-low latency tracking of head and hand movements to prevent user disorientation (cybersickness). This is achieved by fusing:

• Inside-Out Tracking: Cameras on the headset track visual features in the room for positional tracking.
• Inertial Tracking: High-rate IMUs (1000 Hz+) provide immediate responsiveness to rotational movements, filling the gap between camera frames.
• Magnetic/Ultrasonic Tracking: For absolute positional reference or controller tracking. The fusion creates a smooth, jitter-free 6-Degree-of-Freedom (6DoF) pose estimation essential for immersion.

Predictive maintenance systems on factory floors use multi-sensor fusion to monitor the health of critical machinery like motors, pumps, and turbines.

• Vibration Analysis: Accelerometers detect abnormal frequencies indicating bearing wear or imbalance.
• Acoustic Emission: Microphones pick up ultrasonic sounds from cracks or leaks.
• Thermal Imaging: Infrared cameras detect overheating components.
• Current Sensors: Monitor motor electrical signatures for faults. Fusing these data streams provides a holistic health score, reducing false alarms from any single sensor and enabling precise fault diagnosis before catastrophic failure.

Modern smartphones are dense sensor hubs that use fusion to enable features transparently to the user.

• Screen Auto-Rotation: Fusing accelerometer (gravity vector) and gyroscope (rotation rate) to determine device orientation more accurately and quickly than either sensor alone.
• Step Counting/Pedestrian Dead Reckoning: Combining accelerometer peaks with gyroscope data to filter out non-walking motions and estimate direction, improving GPS accuracy in urban canyons.
• Ambient Light & Color Sensing: Fusing data from dedicated light and proximity sensors with camera and screen brightness to automatically adjust display color temperature and brightness for optimal viewing.

A recursive algorithm that uses a series of noisy measurements over time to produce statistically optimal estimates of unknown variables. It is a foundational technique for linear dynamic systems.

• Core Function: Predicts the next state of a system, then updates that prediction with a new measurement, weighting each by their estimated covariance (uncertainty).
• TinyML Relevance: The Extended Kalman Filter (EKF) handles non-linear systems common in robotics and is computationally efficient enough for microcontroller deployment.
• Example: Fusing a noisy GPS position (low frequency, high latency) with a high-frequency accelerometer and gyroscope (IMU) to estimate a vehicle's smooth, accurate position and velocity in real-time.

An electronic device that combines accelerometers and gyroscopes (and often magnetometers) to measure a body's specific force, angular rate, and orientation.

• Primary Sensors:
  • Accelerometer: Measures proper acceleration (g-force + motion).
  • Gyroscope: Measures angular velocity (rate of rotation).
  • Magnetometer: Measures magnetic field strength, often used as a compass.
• Fusion Role: IMU data is high-frequency but drifts over time. It is typically fused with absolute but slower/less frequent sensors (like GPS or vision) to provide stable, drift-free orientation and motion tracking.
• Use Case: Critical for drone flight stabilization, smartphone screen rotation, and pedestrian dead reckoning.

Methods for analyzing sequences of data points indexed in time order to extract statistics, identify patterns, and forecast future values from sensor streams.

• Key Techniques:
  • Trend & Seasonality Decomposition: Separating long-term direction, repeating cycles, and residual noise.
  • Autocorrelation: Measuring how a signal correlates with a delayed version of itself.
  • Forecasting: Using models like ARIMA or LSTMs to predict future sensor readings.
• Fusion Context: Provides the statistical framework for understanding the temporal behavior of individual sensor streams before they are combined, enabling smarter fusion algorithms that account for signal dynamics.

The process of adjusting a sensor's output to match a known standard, correcting for systematic errors like offset, gain, and non-linearity to ensure measurement accuracy.

• Calibration Types:
  • Factory Calibration: Performed by the manufacturer.
  • Field/In-Situ Calibration: Performed in the deployment environment to account for installation effects and aging.
• Fusion Prerequisite: Accurate fusion is impossible with uncalibrated sensors. Errors compound, leading to divergent and unreliable fused estimates. For IMUs, this includes determining biases, scale factors, and axis misalignment.
• TinyML Challenge: On-device calibration routines must be lightweight, often using simple least-squares methods or pre-computed correction matrices stored in flash memory.

A data processing technique where a fixed-size buffer of the most recent samples moves sequentially over a data stream, enabling real-time analysis of the latest segment of time-series data.

• Mechanism: As a new sample arrives, the oldest sample is discarded, and the window "slides" forward by one position.
• Fusion Application: Essential for real-time fusion algorithms that operate on a recent history of data, such as computing short-term Fourier transforms (STFT) for audio-visual fusion or maintaining a buffer of IMU readings for integration.
• Memory Efficiency: For TinyML, the window size is a critical design parameter, directly impacting RAM usage. Circular buffer implementations are standard to avoid costly data shifting.

The process of transforming raw, high-dimensional sensor data into a reduced set of informative, non-redundant values (features) that are more suitable for machine learning model training and inference.

• Common Sensor Features:
  • Statistical: Mean, variance, standard deviation, root mean square (RMS).
  • Temporal: Zero-crossing rate, peak count, signal entropy.
  • Spectral: Band energy from FFT, Mel-frequency cepstral coefficients (MFCCs).
• Fusion Level: Fusion can occur at different levels:
  • Low-Level (Raw Data): Fusing sensor readings directly (e.g., Kalman filter).
  • Feature-Level: Extracting features from each sensor first, then fusing the feature vectors.
  • Decision-Level: Running separate classifiers on each sensor stream and fusing their output probabilities.
• TinyML Optimization: Feature extraction is often the most compute-intensive step, requiring careful optimization of DSP kernels.