Bundle Adjustment

Bundle adjustment is fundamentally a non-linear least squares optimization problem. It minimizes the sum of squared reprojection errors—the differences between observed 2D image points and the projection of estimated 3D points. This is typically solved using iterative algorithms like Levenberg-Marquardt or Gauss-Newton, which linearize the problem around the current estimate at each step. The non-linearity arises from the perspective projection model and, often, complex camera lens distortion parameters.

Unlike sequential or filtering approaches, bundle adjustment simultaneously refines all unknown parameters in a single, cohesive optimization. This includes:

• 3D Structure: The coordinates of all scene points (the 'bundle' of rays).
• Camera Extrinsics: The position and orientation (pose) of each camera.
• Camera Intrinsics: Focal length, principal point, and distortion coefficients. This joint optimization allows errors to be distributed globally, correcting drift and inconsistencies that accumulate in incremental methods.

The Jacobian matrix at the heart of the least squares problem exhibits a characteristic sparse block structure. Each 3D point is observed by only a subset of cameras, and each camera sees only a subset of points. This results in a matrix with mostly empty blocks. Exploiting this sparsity via Schur complement trick (or elimination) is critical for computational efficiency, reducing the problem size from solving a dense system with (millions of points + hundreds of cameras) variables to a much smaller system involving only camera parameters.

Bundle adjustment operates in two primary modes:

• Global (Offline) BA: Processes all images and points in a batch after data capture. It provides the highest possible accuracy and is used in photogrammetry (e.g., creating maps from drone imagery).
• Local (Online) BA: Used in Visual SLAM and Visual Odometry systems. It optimizes only a recent sliding window of camera poses and the points they observe. This maintains real-time performance by limiting problem size while still mitigating drift. Pose-graph optimization often runs alongside to correct longer-term loop closures.

Initial feature matching and triangulation inevitably produce incorrect correspondences (outliers). Standard least squares is highly sensitive to these. Therefore, practical bundle adjustment employs robust cost functions (e.g., Huber loss, Cauchy loss) that reduce the influence of large residuals. This is often combined with pre-filtering using algorithms like RANSAC (Random Sample Consensus) during the feature matching stage to reject gross outliers before optimization begins.

Despite sparsity, large-scale bundle adjustment is computationally intensive. Complexity grows with the number of points and cameras. Key challenges include:

• Memory: Storing the sparse Jacobian and Hessian approximations for millions of points.
• Compute: Solving large linear systems each iteration. This drives the need for efficient linear solvers (like SuiteSparse, Ceres Solver), and motivates approximations like local BA. On-device implementations must carefully manage these constraints.

SLAM is the overarching real-time process where a device builds a map of an unknown environment while simultaneously tracking its own location within it. Bundle adjustment often acts as the global optimization backend in visual SLAM systems, refining the map and trajectory after loop closures.

• Frontend vs. Backend: The frontend (e.g., visual odometry) provides initial pose estimates, while the backend (bundle adjustment) performs global refinement.
• Key Relationship: In systems like ORB-SLAM, bundle adjustment is triggered after loop detection to correct accumulated drift across the entire map.

VIO is a sensor fusion technique that combines a camera and an Inertial Measurement Unit (IMU) to estimate high-frequency, robust device pose. It provides the initial pose estimates and 3D point observations that are later refined by bundle adjustment.

• Complementary Roles: VIO offers real-time, drift-prone local tracking. Bundle adjustment provides offline or less-frequent global optimization using a larger window of frames.
• Tightly-Coupled Systems: Advanced systems perform visual-inertial bundle adjustment, jointly optimizing camera poses, 3D points, IMU biases, and velocity states within a single optimization framework.

Pose graph optimization is a more efficient, simplified cousin of bundle adjustment used in large-scale SLAM. Instead of optimizing both camera poses and all 3D points, it optimizes only the poses, with constraints derived from relative measurements between them.

• Reduced Complexity: By marginalizing out the 3D points, it creates a graph where nodes are poses and edges are constraints (from odometry or loop closures). This scales better for very long trajectories.
• Hierarchical Optimization: In modern systems, pose graph optimization handles large-scale loop closures and trajectory correction, while local bundle adjustment refines a smaller window of recent frames and their associated map points for highest accuracy.

A TSDF is a volumetric representation used for dense 3D surface reconstruction. While bundle adjustment optimizes sparse feature points, the TSDF integrates depth measurements (e.g., from a depth camera) into a globally consistent, dense 3D model.

• Data Fusion: The camera poses optimized by bundle adjustment are used to fuse depth frames into the TSDF volume. Accurate poses are critical to avoid blurry or double surfaces.
• On-Device Use: Efficient implementations like voxel hashing enable real-time TSDF fusion on mobile devices, creating the detailed 3D meshes seen in AR scanning apps.

ICP is an algorithm for aligning two 3D point clouds or a point cloud to a model. It is closely related to bundle adjustment but operates directly on dense geometry rather than sparse features and reprojection error.

• Alignment vs. Optimization: ICP finds a rigid transformation (rotation & translation) to align surfaces. Bundle adjustment finds a non-linear least-squares solution for multiple cameras and points.
• Hybrid Approaches: Visual-inertial systems may use ICP to align local dense depth maps, providing additional constraints for the bundle adjustment problem. Point-to-plane ICP is a common variant that uses surface normals for more accurate alignment.

Loop closure detection is the process of recognizing when a device revisits a known location. It provides the critical constraints that enable bundle adjustment to correct long-term drift accumulated by odometry systems like VIO.

• Triggering Optimization: A successful loop closure provides a strong measurement between the current camera pose and a past pose. This constraint is added to the optimization problem, allowing bundle adjustment to distribute the error correction across the entire trajectory.
• Visual Bag-of-Words: Efficient on-device systems often use a visual vocabulary (e.g., DBoW2) to quickly match current keyframes against a database of past keyframes for loop candidate retrieval.