Structure from Motion (SfM)

The initial stage where distinctive keypoints (like corners or blobs) are identified in each image. Each keypoint is assigned a descriptor, a numerical vector (e.g., SIFT, ORB) that encodes the local image appearance around it. This enables matching the same physical point across different images, even under varying lighting or viewpoint changes.

Descriptors from different images are compared to establish correspondences. Initial matches are often noisy, containing many outliers. Robust estimation algorithms like RANSAC are used with geometric models (e.g., the Fundamental Matrix or Essential Matrix) to verify matches and reject outliers, ensuring only geometrically consistent correspondences proceed. This step is critical for accurate initial pose estimation.

SfM typically begins with a seed pair of images with a verified geometric relationship. Their relative pose is estimated, and the 3D positions (scene points) of their matched features are calculated via triangulation. New images are then incrementally registered to the growing 3D model by solving the Perspective-n-Point (PnP) problem, and new scene points are triangulated from new correspondences.

The core non-linear optimization that refines the entire reconstruction. It jointly adjusts all estimated parameters—3D point coordinates, camera poses, and often camera intrinsics—to minimize the total reprojection error. This is the difference between where a 3D point projects onto an image and where it was actually observed. Bundle adjustment corrects drift and error accumulation from incremental steps.

Following sparse SfM (which produces a point cloud of key features), dense reconstruction techniques generate a complete surface model. Methods like Multi-View Stereo (MVS) compute depth or disparity for (nearly) every pixel by exploiting photo-consistency across multiple registered views. The output is typically a dense point cloud, mesh, or textured 3D model, used in applications like digital twins and heritage preservation.

A critical post-processing step for real-world applications. Pure SfM reconstructs the scene up to an unknown scale factor. Scale is often resolved using known distances in the scene, sensor data from an Inertial Measurement Unit (IMU), or by geo-registering the model to real-world coordinates using GPS tags or known ground control points. This transforms the model into a metrically accurate, georeferenced asset.

Bundle adjustment is the non-linear optimization backbone of SfM. It jointly refines all estimated parameters—3D point positions, camera poses, and often intrinsic calibration—by minimizing the total reprojection error across all images. This global optimization corrects drift and errors accumulated during incremental reconstruction.

• Purpose: Achieves a globally consistent, optimal reconstruction.
• Algorithm: Typically uses the Levenberg-Marquardt algorithm to solve this large, sparse least-squares problem.

SLAM is the real-time counterpart to offline SfM. While both estimate 3D structure and camera pose, SLAM is designed for sequential, online operation by a moving agent (e.g., robot, AR headset). It emphasizes temporal consistency, loop closure detection, and real-time performance.

• Key Difference: SfM often processes unordered image sets offline; SLAM processes sequential frames online.
• Application: Essential for robotics, autonomous vehicles, and real-time augmented reality.

Visual Odometry is the pose estimation component within a SLAM pipeline. It focuses specifically on incrementally estimating the camera's ego-motion (pose change) between consecutive frames. VO provides the local trajectory, which SLAM then integrates into a globally consistent map.

• Function: Estimates relative camera motion.
• Relation to SfM: SfM can be seen as a global, batch-mode version of the pose estimation problem that VO solves sequentially.

Multi-View Stereo is the dense reconstruction stage that typically follows SfM. While SfM produces a sparse 3D point cloud and camera poses, MVS uses those poses to generate a dense surface or depth map for every pixel, creating a complete 3D model.

• Input: Sparse SfM output (cameras, points).
• Output: Dense point cloud, mesh, or textured 3D model.
• Example: Turning photos of a building into a dense, watertight 3D mesh.

Epipolar geometry defines the projective relationship between two views. The fundamental matrix (F) encapsulates this geometry for uncalibrated cameras. It is a core mathematical tool in SfM, used during feature matching and initial pose estimation to constrain corresponding points to lie on epipolar lines.

• Role: Provides geometric constraints for establishing correspondences and estimating relative camera motion.
• Essential Matrix (E): The calibrated version of F, used when camera intrinsics are known.

Photogrammetry is the broader scientific discipline of making measurements from photographs. SfM is a specific, highly automated computational technique within modern digital photogrammetry. Traditional photogrammetry often relied on manually identified points and known camera positions.

• Scope: Includes aerial survey, topographic mapping, and cultural heritage digitization.
• SfM's Impact: Automated and democratized high-accuracy 3D reconstruction from consumer-grade imagery.