Visual-Inertial Odometry (VIO)

VIO algorithms fuse raw Inertial Measurement Unit (IMU) readings directly with visual feature observations in a single, joint optimization framework. Unlike loosely coupled approaches that process camera and IMU data independently, tightly coupled fusion jointly minimizes the reprojection error of visual landmarks and the inertial error of the IMU preintegration. This allows the IMU to provide a high-rate motion prior (often 100-1000 Hz) that constrains the visual feature tracking between frames, making the system robust to aggressive motion and short-term visual occlusions. The state vector typically includes the platform's pose, velocity, and IMU biases.

A fundamental advantage of VIO over monocular visual odometry is the recovery of metric scale. Pure monocular vision suffers from scale ambiguity—it cannot distinguish between a large, distant scene and a small, close one. By fusing visual data with an accelerometer that measures gravity and linear acceleration in metric units (m/s²), VIO renders the scale, as well as the roll and pitch angles relative to the gravity vector, fully observable. This eliminates the scale drift that plagues visual-only systems and provides a globally aligned gravity direction, which is essential for stable navigation.

Advanced VIO systems can estimate and refine calibration parameters in real-time, a feature critical for robust long-term deployment. This includes:

• Temporal offset calibration: Estimating the unknown time delay between the unsynchronized camera and IMU data streams, often to sub-millisecond accuracy.
• Extrinsic calibration: Continuously refining the rigid-body transformation (rotation and translation) between the camera and IMU coordinate frames.
• Intrinsic IMU calibration: Estimating the scale and misalignment of the accelerometer and gyroscope axes. This online self-calibration eliminates the need for costly factory calibration procedures and compensates for mechanical shifts due to temperature or vibration.

VIO provides a graceful degradation of performance rather than catastrophic failure when visual conditions deteriorate. In scenarios with low texture, high dynamic range, or motion blur where visual feature tracking fails, the IMU propagates the state estimate via dead reckoning. The system's uncertainty grows cubically over time during pure inertial propagation, but the moment visual features are reacquired, the optimization loop immediately corrects the drift. This complementary sensing modality makes VIO the preferred solution for applications ranging from fast-moving drones navigating featureless corridors to augmented reality headsets experiencing rapid user head rotations.

To maintain bounded computational complexity for real-time operation, VIO systems employ a sliding window estimator. Instead of optimizing over all historical states (full batch optimization), the system maintains a fixed number of recent keyframes and landmarks. When a new keyframe is added, an old state must be removed. To preserve the information contained in the removed state without double-counting, a mathematical technique called marginalization is applied. This converts the removed state's connections into a dense linear prior on the remaining variables, preserving the system's observability properties and preventing information loss while keeping the optimization problem at a constant size.

The foundational recursive estimator for VIO. A Kalman filter fuses noisy IMU measurements (prediction step) with visual observations (update step) to produce an optimal state estimate. In VIO, variants like the Error-State Kalman Filter (ESKF) are preferred because they linearize around a nominal state, avoiding singularities in 3D rotations and handling the high-frequency IMU data efficiently.

A modern alternative to filtering for VIO backends. Instead of marginalizing past states, factor graphs represent the estimation problem as a bipartite graph of robot poses (variables) and sensor measurements (factors). VIO systems like OKVIS and VINS-Mono use nonlinear least squares (e.g., Levenberg-Marquardt) to solve for the maximum a posteriori trajectory, achieving higher accuracy through iterative relinearization.

A critical computational technique for tightly-coupled VIO. High-rate IMU measurements (100-1000 Hz) between slower camera frames (10-30 Hz) are condensed into a single preintegrated measurement comprising delta position, velocity, and rotation. This prevents the need to reintegrate raw IMU data every time the optimization linearization point changes, dramatically reducing the computational load in factor graph-based systems.

The rigid-body transformation T_cam_imu defining the 6-DOF spatial relationship between the camera and IMU coordinate frames. Accurate extrinsic calibration is non-negotiable for VIO performance. Errors in rotation or translation directly bias the motion estimate. Modern toolkits like Kalibr estimate these parameters jointly with intrinsic camera and IMU parameters using continuous-time batch optimization over a calibration target sequence.

A mathematical assessment of which states a VIO system can uniquely determine. A monocular VIO system has 4 unobservable degrees of freedom: 3 for global position and 1 for yaw (rotation around gravity). Understanding this degeneracy is crucial for filter consistency. The canonical Lie derivative analysis reveals that proper observability-constrained filtering prevents spurious information gain along these unobservable directions, maintaining estimator consistency.

The process of recognizing a previously visited location to correct accumulated drift. VIO alone suffers from unbounded drift over long trajectories. A bag-of-words model like DBoW2 extracts visual features and matches them against a historical database. When a loop is detected, a global constraint is added to the factor graph, enabling pose graph optimization to distribute the error and produce a globally consistent map.