Multi-View Stereo (MVS)

Unlike Structure from Motion (SfM) which establishes sparse feature correspondences, MVS algorithms compute dense pixel-wise correspondences across multiple images. This involves matching every pixel (or a dense sampling of pixels) in a reference image to its corresponding points in neighboring source images. The process is computationally intensive and is the foundation for generating a dense point cloud. Common algorithmic approaches include:

• Patch-based matching: Compares small image patches using metrics like Normalized Cross-Correlation (NCC) or Sum of Squared Differences (SSD).
• Depth map fusion: Independently estimates a depth map for each camera view and then merges them into a unified 3D model.
• Volumetric methods: Operate directly in 3D space using a cost volume to aggregate matching scores.

MVS is not an end-to-end pipeline; it is a dense reconstruction stage that depends on precise prior camera calibration. Essential inputs include:

• Camera intrinsic parameters: Focal length, principal point, and lens distortion coefficients for each camera.
• Camera extrinsic parameters: The precise camera pose (position and orientation) for each image in a shared world coordinate system, typically provided by a preceding Structure from Motion (SfM) pipeline.
• Image sequence: Multiple overlapping images of a static scene, captured from different viewpoints. The accuracy of the final 3D reconstruction is directly tied to the accuracy of these calibration parameters. Errors in camera pose or lens distortion models propagate directly into the 3D geometry, causing artifacts like misaligned surfaces or 'ghosting'.

The core hypothesis of MVS is that a correctly estimated 3D point, when projected into all images where it is visible, should exhibit photometric consistency. This means the color or intensity values of the corresponding pixels should be similar, assuming Lambertian (non-shiny) surfaces and consistent lighting. The reconstruction problem is framed as an optimization that minimizes a photometric error function. For a hypothesized 3D point, the error is computed as the variance or difference between the pixel colors it projects to in the source images. Advanced MVS systems account for:

• Non-Lambertian surfaces: Using robust cost functions or modeling view-dependent appearance.
• Occlusions: Identifying and excluding images where the point is not visible.
• Lighting variation: Applying normalization or more complex radiometric models.

The primary output of an MVS pipeline is a dense, colored 3D point cloud, where each point has an XYZ coordinate and an RGB color value aggregated from the source images. This raw point cloud is often processed further through a surface reconstruction stage to create a continuous, watertight polygon mesh. Key post-processing steps include:

• Point cloud filtering: Removing statistical outliers and noise.
• Poisson Surface Reconstruction: An algorithm that creates a mesh by solving for an implicit surface (like a Signed Distance Function) that best fits the oriented point cloud.
• Mesh simplification and texturing: Reducing polygon count for efficiency and projecting source images onto the mesh to create a texture map for realistic rendering. This final mesh is the standard deliverable for applications in digital heritage, visual effects, and digital twin creation.

MVS must resolve several fundamental ambiguities inherent in inferring 3D from 2D images. Key challenges and mitigation strategies include:

• Textureless regions: Areas with uniform color (e.g., white walls) provide no visual cues for matching. Solutions often involve using global smoothness priors (assuming depth varies smoothly) or integrating data from other sensors.
• Occlusions: Points visible in one image may be hidden in another. Robust MVS algorithms perform visibility testing to only compare pixels in images where the 3D point is unoccluded.
• Reflective or transparent surfaces: These violate the photometric consistency assumption. Advanced methods may use polarization cues or learned models to handle these materials.
• Depth discontinuities: At object boundaries, depth changes abruptly. Algorithms use techniques like adaptive window matching or semantic segmentation to preserve sharp edges.

Traditional MVS relies on hand-crafted similarity measures and optimization. Learning-based MVS has become dominant, using convolutional neural networks (CNNs) to learn matching costs and regularization from data. The standard architecture involves:

• Feature extraction: A CNN extracts deep feature maps from each input image.
• Cost volume construction: Features are warped into a range of hypothesized depth planes to build a 3D cost volume, where each cell represents the matching cost for a specific depth at a specific pixel.
• Cost volume regularization: A 3D CNN aggregates context across neighboring pixels and depth hypotheses to produce a refined probability volume.
• Depth map prediction: The final depth map is computed via soft argmin or winner-takes-all on the probability volume. Frameworks like MVSNet and its successors (e.g., PatchmatchNet, CasMVSNet) demonstrate superior performance on textured and textureless regions compared to classical methods.

Structure from Motion (SfM) is the essential prerequisite for MVS. It is a computer vision technique that takes a collection of unordered 2D images and simultaneously estimates:

• The 3D structure of the scene as a sparse point cloud.
• The camera poses (position and orientation) for each input image.

SfM provides the calibrated camera parameters and an initial geometric scaffold. MVS then uses this precise calibration to perform dense reconstruction, converting the sparse SfM output into a complete surface.

Bundle adjustment is the non-linear optimization backbone of both SfM and MVS pipelines. It refines all estimated parameters—3D point coordinates, camera poses, and lens distortion—to minimize the total reprojection error. This error is the pixel-distance between where a 3D point should project into an image and where its corresponding 2D feature was actually detected. By solving this large optimization problem, bundle adjustment ensures the multi-view geometry is globally consistent and accurate, which is critical for high-quality MVS results.

Depth map estimation is the fundamental per-image operation within an MVS pipeline. For each calibrated input image, the algorithm computes a depth map—a 2D image where each pixel value represents the distance from that camera to the visible scene surface.

MVS algorithms typically work by:

1. Estimating a depth (and often a normal map) for every reference image.
2. Fusing these per-view depth maps into a unified global 3D representation, such as a point cloud or a Truncated Signed Distance Function (TSDF) volume. This fusion step resolves inconsistencies and fills holes.

Surface reconstruction is the final stage that follows MVS. While MVS produces a dense but unstructured point cloud, surface reconstruction infers a continuous, watertight surface.

This process involves:

• Filtering the noisy point cloud.
• Estimating a surface from the points, often using algorithms like Poisson reconstruction or Screened Poisson.
• Generating a polygonal mesh (vertices and faces) that represents the object's skin.
• Optionally, performing texture mapping by projecting the original images onto the mesh to create a photorealistic 3D model.

Photogrammetry is the overarching science and technology of obtaining reliable measurements from photographs. MVS is a core, modern computational technique within the photogrammetry workflow.

A standard photogrammetry pipeline is:

1. Image Acquisition (capturing photos with sufficient overlap).
2. Structure from Motion (SfM) (sparse reconstruction and camera calibration).
3. Multi-View Stereo (MVS) (dense reconstruction).
4. Surface Reconstruction & Texturing (mesh generation).

Thus, MVS is the high-fidelity geometric heart that turns calibrated photos into dense 3D data.

Visual SLAM (Simultaneous Localization and Mapping) shares core principles with SfM/MVS but operates under different constraints. While traditional MVS assumes a pre-captured set of images of a static scene, Visual SLAM performs online, sequential reconstruction in real-time.

Key comparisons:

• Goal: SLAM focuses on localization (tracking the camera) for navigation, with mapping as a byproduct. MVS focuses on high-quality, offline mapping.
• Input: SLAM processes a live video stream. MVS processes a curated set of images.
• Output: SLAM often produces a sparse or semi-dense map for tracking. MVS aims for a complete, dense, production-quality model.
• Optimization: SLAM uses local bundle adjustment or filtering for speed; MVS uses global, computationally intensive optimization for accuracy.