Structure from Motion (SfM)

The primary output of a classic SfM pipeline is a sparse point cloud. It reconstructs only the 3D positions of distinct, trackable feature points (like SIFT or SURF keypoints) across multiple images. This is computationally efficient for initial camera pose estimation and scene geometry but lacks surface detail.

• Output: A set of 3D points and the estimated camera positions from which they were observed.
• Purpose: Serves as the foundational geometric scaffold for subsequent dense reconstruction steps like Multi-View Stereo (MVS).

SfM pipelines are categorized by their optimization strategy.

• Incremental SfM: Starts with a two-view reconstruction and incrementally adds new images, performing Bundle Adjustment after each addition. It is robust but can accumulate drift and is computationally heavy for large datasets. Tools like COLMAP often use this approach.
• Global SfM: First estimates all camera rotations, then translations, and finally performs a single global bundle adjustment. It is faster and avoids sequential drift but can be less stable if pairwise geometry estimates are noisy.

SfM's core mechanics are feature matching and triangulation.

• Feature Matching: Identifies correspondences of the same physical point across different images. This is often done via descriptor matching (e.g., SIFT) and robust estimation with RANSAC to filter outliers.
• Triangulation: For a matched feature point seen in two or more images with known camera poses, its 3D position is calculated as the intersection of the sight lines (rays) from each camera center through the image point. Accuracy increases with a larger baseline (distance between camera positions).

Bundle Adjustment (BA) is the non-linear least-squares optimization that refines the 3D coordinates of scene points and the parameters (pose, focal length, distortion) of the cameras simultaneously. It minimizes the reprojection error—the difference between the observed 2D feature locations and the projected 3D points back onto the image planes.

• Function: BA is what makes SfM accurate, correcting errors from noisy initial estimates.
• Scale: It is the most computationally intensive step, often leveraging sparse linear algebra solvers like those in the Ceres Solver or g2o libraries.

A fundamental characteristic of SfM from unordered images is that it reconstructs the scene up to an unknown scale factor. The model is metrically correct in shape but not in absolute size. Scale can be recovered by:

• Including an object of known dimensions in the scene.
• Using sensor data (e.g., IMU readings for velocity) in a Visual-Inertial Odometry (VIO) system.
• Knowing the camera's intrinsic parameters precisely (e.g., focal length in millimeters).

This distinguishes pure SfM from LiDAR or RGB-D sensing, which provide metric scale directly.

SfM is often conflated with Visual SLAM; key differences are in context and output.

• SfM: Typically applied to offline processing of unordered image collections (e.g., tourist photos). Focus is on accurate, globally consistent 3D models. It often assumes a static scene.
• Visual SLAM: Designed for online, sequential processing from a moving camera (e.g., on a robot). Focus is on real-time localization and incremental mapping for navigation. It must handle dynamic scenes and include loop closure to correct drift.

While they share algorithms (feature tracking, BA), SfM is reconstruction-centric, and SLAM is localization-centric.

Photogrammetry is the broader science of making measurements from photographs. SfM is a specific computational technique within this field. The core principle is triangulation: determining the 3D location of a point by measuring it from two or more known camera positions.

• Key Input: Overlapping 2D images.
• Core Output: Dense 3D point clouds, textured meshes, and orthomosaics.
• Primary Use Cases: Surveying, mapping, cultural heritage documentation, and creating digital elevation models (DEMs).
• Relationship to SfM: SfM automates and scales the traditional photogrammetric pipeline, solving for camera poses and sparse structure simultaneously from unordered image collections.

Simultaneous Localization and Mapping (SLAM) is a real-time process where an agent (like a robot or AR device) builds a map of an unknown environment while simultaneously tracking its own location within it. It is closely related to SfM but differs in key operational constraints.

• Key Distinction: SLAM is online and sequential, processing sensor data (often from cameras, LiDAR, or IMUs) frame-by-frame as it's captured. SfM is typically offline and global, processing a complete set of images after capture.
• Sensor Fusion: Modern visual SLAM (vSLAM) often fuses camera data with inertial measurement units (IMUs) for robustness.
• Primary Use Cases: Autonomous robot navigation, augmented reality, and real-time 3D scanning.

Multi-View Stereo (MVS) is the dense reconstruction stage that typically follows SfM. While SfM produces a sparse 3D point cloud and camera poses, MVS generates a dense point cloud or a complete surface mesh.

• Input: The calibrated camera poses and sparse points from SfM, plus the original images.
• Process: For each pixel in multiple images, MVS algorithms search for correspondences across views to compute highly detailed depth maps, which are fused into a dense model.
• Algorithm Examples: PatchMatch Stereo, Plane Sweeping, and Depthmap Fusion.
• Output: Dense, photo-realistic 3D models suitable for visualization, inspection, and measurement.

Bundle Adjustment is the non-linear optimization backbone of SfM and photogrammetry. It refines the 3D coordinates of scene geometry (the points) and the parameters (position, orientation, focal length) of the cameras simultaneously to minimize the total reprojection error.

• Reprojection Error: The distance between a measured 2D image point and the projection of its estimated 3D point back into the image.
• Mathematical Formulation: A large-scale sparse least squares problem, often solved using the Levenberg-Marquardt algorithm.
• Role in SfM: Applied incrementally during reconstruction and as a final global refinement step to achieve highly accurate, globally consistent models.

Visual Odometry (VO) is the process of estimating the ego-motion of an agent (e.g., a vehicle or robot) by analyzing the associated camera images. It is a core component of visual SLAM but can operate independently for short-term pose tracking.

• Process: VO typically works by tracking feature points across consecutive image frames to estimate incremental camera movement.
• Scale Ambiguity: Pure monocular VO estimates motion up to an unknown scale factor. Scale can be resolved using sensor fusion (e.g., with an IMU) or known physical dimensions in the scene.
• Relationship to SfM: VO is the sequential, local counterpart to SfM's global optimization. SfM can use VO techniques for initial pose estimation within an image sequence.

Neural Radiance Fields (NeRF) is a novel view synthesis technique that uses a fully-connected deep neural network to represent a continuous volumetric scene. Given a set of input images with known camera poses (often from SfM), it learns to map a 3D location and viewing direction to a color and density.

• Input: A set of posed images (camera calibration from SfM is critical).
• Core Innovation: Represents a scene as a continuous function, enabling high-fidelity rendering of novel viewpoints with complex view-dependent effects like specularities.
• Contrast with SfM/MVS: While SfM/MVS produce explicit geometric representations (point clouds, meshes), NeRF produces an implicit representation. The fields are complementary: SfM provides the necessary camera poses for NeRF training, and NeRF can generate novel training views for SfM pipelines.