Sensor Fusion

The Kalman Filter (KF) is an optimal recursive estimator for linear systems with Gaussian noise. It operates in a two-step predict-update cycle:

• Predict: Projects the current state estimate forward in time using the system's dynamic model.
• Update: Corrects the prediction with a new sensor measurement, weighted by the estimated uncertainty (covariance) of both the prediction and the measurement.

It is foundational for fusing data from sensors like IMUs and wheel encoders. The Extended Kalman Filter (EKF) linearizes non-linear models for broader application, such as fusing GPS with IMU data for localization.

A Particle Filter is a sequential Monte Carlo method used for state estimation in non-linear, non-Gaussian systems. It represents the probability distribution of the state using a set of random samples called particles.

• Each particle is a hypothesis of the true system state.
• The algorithm repeatedly predicts particle motion, weights particles based on how well their predicted sensor readings match actual observations, and resamples to focus computational resources on the most likely hypotheses.

This technique is highly effective for complex problems like robot localization (e.g., Monte Carlo Localization) where the state space is multi-modal or involves complex sensor models like lidar.

The Complementary Filter is a simple, computationally efficient frequency-domain fusion technique. It combines two sensors with complementary noise characteristics: one accurate in the long term (low-frequency) and one accurate in the short term (high-frequency).

• A classic example is fusing an accelerometer (good for low-frequency tilt estimation but noisy due to motion) with a gyroscope (good for high-frequency angular rate but drifts over time).
• The filter uses a high-pass filter on the gyro data and a low-pass filter on the accelerometer data, then sums the outputs. The cutoff frequency parameter balances trust between the sensors.

It is ubiquitous in lightweight applications like stabilizing drones and estimating orientation for consumer devices.

Fusion can occur at different levels of abstraction, trading off information richness for complexity:

• Sensor-Level (Low-Level) Fusion: Raw data from multiple sensors is combined directly before any feature extraction. Example: Combining raw voltage readings from multiple strain gauges. This preserves maximal information but is sensitive to noise and misalignment.
• Feature-Level (Mid-Level) Fusion: Each sensor's data is processed independently to extract features (e.g., edges from a camera, line segments from lidar). These feature vectors are then fused. Example: A visual SLAM system fusing visual feature points with lidar-derived point clouds. This is the most common approach for perception systems.
• Decision-Level (High-Level) Fusion: Each sensor subsystem makes an independent decision or classification (e.g., "object is a car"). These decisions are then fused using techniques like voting or Bayesian inference. This is robust to sensor failure but loses low-level correlation information.

Probabilistic Graphical Models (PGMs), such as Bayesian Networks and Factor Graphs, provide a powerful framework for representing the complex conditional dependencies between states and sensor measurements over time.

• They explicitly model how the state (e.g., robot pose) evolves and how each sensor observation is generated from that state.
• Inference algorithms (e.g., belief propagation) run on these graphs to compute the posterior distribution of the state given all sensor data.
• This formalism underlies modern Simultaneous Localization and Mapping (SLAM) systems, where the graph includes factors for odometry constraints, landmark observations from cameras/lidar, and loop closures.

Deep Learning models, particularly multi-modal neural networks, can learn complex, non-linear fusion functions directly from data without explicit hand-designed models.

• Early Fusion: Raw or processed data from different sensors (e.g., camera pixels and lidar point clouds) are concatenated and fed into a single neural network.
• Late Fusion: Separate neural network branches process each sensor modality, and their high-level embeddings are combined at a later stage.
• Cross-Modal Attention: Mechanisms like transformers learn to dynamically weight and correlate features across sensor modalities (e.g., focusing visual attention on regions indicated by radar returns).

This data-driven approach is key for Vision-Language-Action Models (VLAs) and advanced multi-sensor perception stacks in self-driving cars, where defining explicit fusion rules for raw pixels and point clouds is intractable.

State estimation is the process of inferring the internal, often hidden, state of a dynamical system (e.g., a robot's position, orientation, and velocity) from a sequence of noisy sensor measurements and known control inputs. It is the primary goal of many sensor fusion pipelines.

• Core Algorithms: Kalman Filters (linear), Extended Kalman Filters (non-linear), and Particle Filters are foundational.
• Inputs: Fused sensor data (from IMU, GPS, odometry).
• Output: A probabilistic belief about the system's current state.

Multi-modal perception refers to a system's ability to interpret and understand the environment by processing and correlating data from different sensory modalities, such as vision, lidar, radar, and touch. Sensor fusion is the technical implementation that enables this capability.

• Modality Examples: Visual cameras (texture, color), LiDAR (precise geometry), Radar (velocity, works in fog/rain), Tactile sensors (force, contact).
• Key Benefit: Overcomes the limitations of any single sensor type (e.g., camera failure in low light, LiDAR failure in heavy rain).

A Kalman Filter is an optimal recursive algorithm for estimating the state of a linear dynamical system from a series of incomplete and noisy measurements. It is the most widely used algorithm for low-level sensor fusion (e.g., fusing IMU and GPS).

• Two-Step Process: Prediction (projects state forward using a model) and Update (corrects prediction with new sensor data).
• Core Output: A Gaussian distribution representing the estimated state and its covariance (uncertainty).
• Variants: Extended Kalman Filter (EKF) handles non-linear systems; Unscented Kalman Filter (UKF) often provides better non-linear performance.

Sensor calibration is the prerequisite process of determining the intrinsic parameters (e.g., focal length, distortion) and extrinsic parameters (position and orientation relative to a common frame) of each sensor in a multi-sensor suite. Accurate fusion is impossible without precise calibration.

• Intrinsic Calibration: Corrects for sensor-specific biases and noise models (e.g., camera lens distortion, IMU bias).
• Extrinsic Calibration: Determines the rigid transformation (translation and rotation) between sensors (e.g., camera-to-lidar alignment).
• Ongoing Process: May require periodic re-calibration due to mechanical shifts or thermal effects.

Uncertainty quantification is the practice of measuring and representing the confidence or error bounds associated with a model's predictions or a system's estimated state. In sensor fusion, it is critical for weighting sensor inputs and assessing reliability.

• Represents: Sensor noise, model inaccuracies, and environmental ambiguity.
• Mathematical Forms: Often expressed as a covariance matrix in filtering approaches.
• Downstream Use: Informs decision-making; a system may enter a safe fallback mode if fused state uncertainty exceeds a threshold.

The perception stack is the complete software architecture responsible for processing raw sensor data into a structured, actionable understanding of the environment. Sensor fusion is a core middleware layer within this stack.

• Typical Data Flow:
  1. Sensor Drivers: Raw data ingestion.
  2. Pre-processing: Noise filtering, timestamp synchronization.
  3. Feature Extraction/Object Detection: Identifying entities (e.g., using a neural network on camera images).
  4. Sensor Fusion: Combining detections/features across sensors.
  5. Tracking & Prediction: Maintaining object identities and estimating future states.
• Output: A unified world model for the planning and control systems.