Sensor Fusion

Sensor fusion is fundamentally a Bayesian inference problem. It treats all sensor measurements and the system's state as probability distributions, not single values. Core algorithms like the Kalman filter and particle filter recursively apply Bayes' theorem to update the belief about the state (the posterior) as new, noisy data arrives. This explicitly models uncertainty via covariance matrices, allowing the system to weigh reliable sensors more heavily and gracefully handle conflicting or missing data.

A foundational prerequisite for fusion is bringing all sensor data into a common spatiotemporal frame. This involves:

• Time Synchronization: Aligning sensor timestamps to a monotonic clock (e.g., using PTP or hardware triggers) to correct for differing latencies and sampling rates.
• Spatial Calibration: Determining the precise extrinsic parameters (3D position and orientation) of each sensor relative to a body frame, often via target-based or motion-based calibration.
• Temporal Interpolation: For asynchronous sensors, predicting measurements to a common fusion time horizon using process models. Failure here introduces systematic errors that cannot be corrected by downstream algorithms.

Effective fusion exploits the distinct noise characteristics and failure modes of different sensor modalities.

• Complementary Sensors fill each other's perceptual gaps. Example: An IMU provides high-frequency angular velocity and linear acceleration but drifts; a camera provides drift-free but low-frequency, feature-sparse pose updates. Together in Visual-Inertial Odometry (VIO), they provide robust tracking.
• Redundant Sensors (e.g., multiple cameras, GPS + UWB) provide fault tolerance. Fusion algorithms use statistical consistency checks (e.g., Mahalanobis distance) to identify and reject faulty sensor data (outlier rejection), maintaining system integrity.

This defines the level at which raw data is integrated.

• Tightly-Coupled Fusion: Raw, low-level sensor data is fused directly. Example: LiDAR-Inertial Odometry (LIO) where raw IMU readings and LiDAR point cloud constraints are optimized in a single factor graph. This maximizes information usage and accuracy but is computationally complex and less modular.
• Loosely-Coupled Fusion: Each sensor subsystem first produces an independent state estimate (e.g., a pose from a visual odometry module). These higher-level estimates are then fused. This is more modular and simpler but can discard useful information and propagate errors from intermediate processing stages.

The observability of a state (e.g., a robot's global yaw) depends on current sensor inputs and motion. A core characteristic of advanced fusion is managing these changing conditions.

• During degenerate motion (e.g., a car driving straight), certain states become unobservable, causing uncertainty to grow.
• Fusion systems must detect these conditions and appropriately inflate uncertainty estimates or apply constraints. For example, a GPS-INS system knows velocity is highly observable from the IMU, but absolute position is only observable when a GPS fix is available. The Kalman filter covariance matrix dynamically reflects this changing information landscape.

Real-world sensors fail. Robust fusion systems implement multiple defensive layers:

• Pre-Filtering: Algorithms like RANSAC are used within a sensor's processing pipeline to reject spurious measurements before fusion.
• Innovation Gating: Within filters like the Kalman filter, the Mahalanobis distance of a new measurement is computed. If it exceeds a statistical threshold, the measurement is rejected as an outlier.
• Sensor Health Monitoring: Tracking long-term metrics like innovation consistency or received signal strength to detect and de-weight failing sensors (e.g., a camera blinded by sun, a GPS under foliage).
• Degraded Mode Operation: The system should provide a best-effort estimate (e.g., using dead reckoning) when primary sensors fail entirely.

AR/VR headsets require precise, low-latency tracking of the user's head pose. This is achieved by fusing:

• Inertial Measurement Unit (IMU) data (accelerometer, gyroscope) for high-frequency, short-term motion prediction.
• Inside-out tracking cameras that observe the environment to correct for IMU drift (Visual-Inertial Odometry).
• For room-scale VR, external lighthouse or ultra-wideband signals provide absolute positional anchors. Fusion creates the stable, drift-free virtual environment necessary to prevent user disorientation and simulator sickness.

Drones for delivery, inspection, and cinematography depend on fusion for stable flight and navigation.

• GPS provides global position but is unavailable indoors or unreliable near structures.
• IMUs provide attitude and acceleration data for the flight controller's stabilization loops.
• Downward-facing cameras and ultrasonic sensors provide visual odometry and altitude data for precise hovering and landing without GPS.
• Obstacle avoidance systems fuse forward-facing stereo vision, LiDAR, or time-of-flight sensors to create a real-time 3D map for path planning.

Even non-autonomous vehicles use fusion for safety features:

• Adaptive Cruise Control (ACC): Primarily uses radar for precise distance and relative speed to the car ahead. A forward-facing camera may fuse data to classify the object as a car, truck, or motorcycle, informing following distance logic.
• Automatic Emergency Braking (AEB): Fuses radar and camera data to achieve high-confidence object detection and trajectory prediction, reducing false positives (e.g., braking for a roadside sign).
• Lane Keep Assist: Uses the camera to detect lane markings, fused with vehicle dynamics sensors (steering angle, yaw rate) to provide smooth, corrective steering torque.

State estimation is the overarching process of inferring the internal, often unmeasured, variables (the state) of a dynamic system from a sequence of noisy sensor measurements. It is the goal that sensor fusion serves.

• The state typically includes quantities like position, velocity, orientation, and bias terms.
• Sensor fusion is the primary methodology used to achieve accurate state estimation by combining multiple data sources.
• This is a recursive Bayesian inference problem, where the system maintains a probability distribution over possible states (the belief) that is updated as new data arrives.

The Kalman filter is the foundational optimal recursive algorithm for state estimation in linear Gaussian systems. It operates in a two-step predict-update cycle.

• Prediction: Uses a process model to project the state and its uncertainty forward in time.
• Update: Corrects the prediction using a new sensor measurement and a measurement model, weighting the new data based on its relative uncertainty.
• It optimally combines predictions and measurements to produce a minimum mean-square error estimate. Its variants (EKF, UKF) handle nonlinearities.

A factor graph is a bipartite graphical model used to represent the factorization of a complex probability distribution. It is the dominant framework for modern, large-scale smoothing-based state estimation problems like SLAM.

• Variables represent unknown states (poses, landmarks).
• Factors represent probabilistic constraints between variables, derived from sensor measurements (e.g., an odometry factor) or prior knowledge.
• Inference is performed by solving for the variable values that maximize the product of all factors (MAP estimation). This structure makes adding new sensors or loop closures computationally efficient.

Visual-Inertial Odometry (VIO) is a specific, high-performance sensor fusion application that tightly couples a camera and an Inertial Measurement Unit (IMU) to estimate a robot's 3D pose and velocity.

• The camera provides rich, drift-free orientation and position constraints but can fail during fast motion or poor lighting.
• The IMU provides high-frequency acceleration and angular velocity, excellent for tracking rapid motion but drifts quickly due to bias and noise integration.
• Tightly-coupled fusion in VIO uses the IMU data within the process model and directly incorporates visual feature measurements, allowing each sensor to compensate for the other's weaknesses. Examples include VINS-Mono and OKVIS.

This distinction defines the architectural level at which sensor data is combined.

• Tightly-Coupled Fusion: Raw or low-level sensor data are fused within a single estimation framework. Example: feeding raw IMU readings and visual feature tracks directly into a VIO filter. Advantages: Higher accuracy, robustness to individual sensor failure. Disadvantages: Increased complexity, system is sensor-specific.
• Loosely-Coupled Fusion: Each sensor or modality first produces an independent state estimate (e.g., a pose from a visual odometry module, a pose from a GPS receiver). These estimates are then fused at the state level. Advantages: Modular, easier to debug. Disadvantages: Potentially lower accuracy, errors can cascade.

These are critical prerequisites without which sensor fusion will fail. They ensure data is physically and temporally aligned.

• Sensor Calibration: Determines the intrinsic parameters (e.g., camera focal length, distortion) and extrinsic parameters (the precise 6-DOF transform) between sensor frames. A miscalibrated extrinsics matrix will cause fusion to interpret data in the wrong coordinate frame.
• Time Synchronization: Aligns the timestamps of all sensors to a common clock (e.g., using PTP or hardware triggers). A mis-synchronization of even milliseconds can cause severe errors when fusing high-rate IMU data with camera images. Sensor drivers must publish accurately timestamped messages.