Photogrammetry

Structure from Motion is the foundational algorithmic pipeline in photogrammetry that solves the 'chicken-and-egg' problem of 3D reconstruction. It simultaneously estimates the 3D structure of a scene (a sparse point cloud) and the camera poses (position and orientation) from a collection of unordered 2D images.

• Key Process: It begins with feature detection and matching across images, then uses epipolar geometry and triangulation to initialize a 3D scene.
• Output: Produces a sparse point cloud and a set of calibrated camera parameters.
• Example: Reconstructing the 3D layout of an archaeological site from a tourist's photo collection.

Multi-View Stereo is the dense reconstruction stage that follows SfM. Using the known camera poses from SfM, MVS algorithms generate a dense 3D geometry (a point cloud or depth map for every pixel) by finding correspondences across all images.

• Core Function: It turns the sparse SfM output into a dense, usable 3D model.
• Techniques: Includes methods like patch-based stereo, plane-sweeping, and depth-map fusion.
• Contrast with SfM: While SfM solves for camera motion and a sparse set of points, MVS solves for depth at every pixel, given that motion.

Bundle Adjustment is the non-linear optimization backbone of photogrammetry and SfM. It refines the initial estimates of 3D point positions, camera poses, and lens distortion parameters to minimize the total reprojection error.

• Objective Function: Minimizes the sum of squared distances between observed 2D image features and the re-projected 3D points.
• Importance: It is the final, crucial step that ensures global consistency and high accuracy across the entire reconstruction.
• Scale: Often the most computationally expensive step, solved using sparse variants of the Levenberg-Marquardt algorithm.

Visual Simultaneous Localization and Mapping is the real-time, sequential cousin of offline photogrammetry. While photogrammetry often processes batches of images, Visual SLAM processes a live video stream to incrementally build a map of an environment while simultaneously tracking the camera's pose within it.

• Key Difference: SLAM is causal (uses only past frames) and must be highly efficient for real-time operation (e.g., on robots, AR headsets).
• Photogrammetry Link: Many core techniques are shared, including feature tracking, triangulation, and bundle adjustment (often called pose-graph optimization in SLAM).
• Use Case: The core technology enabling autonomous drone navigation or AR apps that understand your living room.

Dense Reconstruction is the overarching goal of generating a complete, continuous surface model from imagery. It encompasses the output of MVS and the subsequent steps of converting a dense point cloud into a usable mesh.

• Pipeline Stages: 1. Depth Map Estimation (per pixel), 2. Depth Map Fusion into a unified volume (e.g., using a Truncated Signed Distance Function - TSDF), 3. Surface Extraction (e.g., via Marching Cubes).
• Output Types: Can produce watertight meshes, textured models, or volumetric representations.
• Contrast: Differs from semantic reconstruction, which adds object labels, and neural reconstruction (like NeRF), which uses an implicit neural representation.

Inverse Rendering is a more ambitious extension of photogrammetry. While classic photogrammetry recovers 3D geometry, inverse rendering aims to decompose a scene into its fundamental physical properties: geometry, material reflectance (BRDF), and lighting.

• Analogy: If rendering is a function f(geometry, material, light) = image, inverse rendering seeks to find the inverse f'(image) = {geometry, material, light}.
• Connection to Photogrammetry: Photogrammetry is often a first step, providing the geometry. Inverse rendering then uses that geometry under different lighting/viewing conditions to solve for materials and lights.
• Application: Critical for creating photorealistic digital twins that behave correctly under novel illumination, enabling true visual fidelity for simulation and design.