Multi-View Stereo (MVS)

MVS requires a set of overlapping 2D images of a static scene captured from known, distinct viewpoints. The camera calibration (intrinsic parameters like focal length) and camera poses (extrinsic parameters: position and orientation) for each image are prerequisites, typically obtained from a preceding Structure from Motion (SfM) pipeline. The quality and coverage of these images directly determine reconstruction completeness.

The fundamental operation is establishing pixel correspondences—finding the same physical 3D point across multiple images. Algorithms match features or compare photometric consistency (color/intensity) along epipolar lines. Once correspondences are found, triangulation is used: the known camera poses and the matched 2D pixel locations are used to calculate the precise 3D world coordinates of the point through geometric intersection.

Unlike SfM, which produces a sparse point cloud of distinctive features, MVS generates a dense reconstruction. The output is typically a dense point cloud with millions of points or a set of per-image depth maps, where each pixel is assigned a distance value. This dense output captures fine surface geometry and texture, suitable for creating watertight meshes.

• Density: Covers textureless regions via photometric or geometric propagation.
• Accuracy: Directly tied to baseline (distance between cameras) and calibration precision.

MVS algorithms heavily depend on the Lambertian assumption—that a surface point appears the same from all viewing angles (non-reflective, non-specular). They optimize for photometric consistency, minimizing the difference in color/intensity of a projected 3D point across all images that observe it. Surfaces that violate this assumption (e.g., glass, mirrors, shiny metal) cause failures, resulting in holes or noise in the reconstruction.

MVS is computationally demanding. Searching for correspondences across high-resolution images and storing per-pixel or per-patch depth hypotheses requires significant processing power and memory. Modern approaches use:

• Patch-based matching: Comparing small image windows.
• Depth-map fusion: Generating and merging depth maps from pairs or groups of views.
• GPU acceleration: Essential for practical runtime on real-world datasets.

MVS vs. Structure from Motion (SfM): SfM solves for camera poses and a sparse point cloud. MVS assumes poses are known and produces dense geometry.

MVS vs. Stereo Vision: Classic stereo uses exactly two calibrated cameras. MVS generalizes this to N cameras (N>2), improving robustness and coverage.

MVS vs. Neural Radiance Fields (NeRF): NeRF learns a continuous volumetric function for view synthesis. Traditional MVS outputs explicit geometry (points/mesh). Newer methods like NeRF can be seen as a differentiable, learned form of MVS.

Structure from Motion is the preceding, complementary pipeline to MVS. SfM solves for the sparse 3D geometry (a point cloud) and the camera poses (position and orientation) from a set of unordered 2D images. It establishes the geometric framework—the 'motion' and initial 'structure'—that MVS requires to perform dense reconstruction.

• Output: Sparse point cloud and camera calibration.
• Key Algorithm: Bundle Adjustment, which jointly optimizes 3D points and camera parameters to minimize reprojection error.
• Relation to MVS: SfM provides the essential camera parameters and an initial alignment that MVS algorithms use to perform dense matching and reconstruction.

Bundle Adjustment is a non-linear optimization backbone for both SfM and MVS. It refines the 3D coordinates of reconstructed points, camera poses, and intrinsic parameters (like focal length) to minimize the total reprojection error—the difference between the observed 2D image points and the projected 3D points.

• Core Function: Global consistency optimization.
• Mathematical Basis: Typically solved via Levenberg-Marquardt algorithm on a large, sparse system of equations.
• Impact: Critical for achieving accurate, drift-free reconstructions over long image sequences or large scenes.

Depth Estimation is the fundamental per-pixel task that MVS performs at scale. For a given pixel in a reference image, MVS algorithms estimate its distance from the camera by finding its correspondence in overlapping neighbor views and applying triangulation.

• Output: A depth map for each input image.
• Methods Range: From traditional patch-based matching (e.g., using Normalized Cross-Correlation) to modern deep learning models that predict depth directly.
• Challenge: The correspondence problem is ill-posed in textureless regions, leading to holes or noise in the reconstruction.

Surface Reconstruction is the subsequent processing step that converts the raw, noisy output of MVS—a set of 3D points or depth maps—into a continuous, usable surface model. MVS provides the data; surface reconstruction creates the final 3D asset.

• Common Output Formats: Triangle meshes (e.g., .obj, .ply) or implicit surfaces like Signed Distance Fields (SDF).
• Algorithms: Include Poisson reconstruction, Delaunay triangulation, and marching cubes.
• Purpose: Enables applications in rendering, simulation, and 3D printing that require watertight, manifold geometry.

Neural Radiance Fields (NeRF) represent a paradigm shift from explicit geometry reconstruction (like MVS) to implicit, neural scene representation. Instead of outputting a point cloud or mesh, a NeRF model a scene as a continuous volumetric function—mapping a 3D location and viewing direction to color and density—using a multilayer perceptron.

• Key Difference: NeRFs are view synthesis models first, capable of generating photorealistic novel views, with geometry (like an SDF) often derived as a secondary output.
• Data Input: Similar to MVS—multiple posed images of a static scene.
• Trade-off: NeRFs typically require longer optimization times but can model complex view-dependent effects and fine details that challenge traditional MVS.

Photogrammetry is the overarching science and technology of obtaining reliable measurements and 3D information from photographs. MVS is a modern, algorithmic realization of photogrammetry principles, automated through computer vision.

• Core Principle: Triangulation from multiple views.
• Historical vs. Modern: Traditionally a manual, specialist process; now largely automated via SfM and MVS pipelines.
• Applications: Extends far beyond MVS to include topographic mapping, cultural heritage digitization, and forensic analysis. MVS is the engine for dense 3D reconstruction within the photogrammetry workflow.