Monocular Depth Estimation

The traditional approach where models are trained on datasets with ground-truth depth maps, typically captured by LiDAR or structured light sensors (e.g., KITTI, NYU Depth V2). Models learn a direct regression from image pixels to depth values.

• Key Loss Functions: L1/L2 loss, Scale-Invariant Logarithmic (SILog) loss, and BerHu loss are used to penalize prediction errors.
• Architectural Evolution: Early models used simple convolutional neural networks (CNNs). The field advanced with encoder-decoder architectures (e.g., U-Net) for multi-scale feature fusion and, more recently, Vision Transformers (ViTs) and convolutional vision transformers (CViTs) that capture long-range dependencies.
• Limitation: Heavy reliance on expensive, often sparse, and sensor-specific ground-truth data, which limits scalability.

A dominant modern paradigm that eliminates the need for ground-truth depth by using photometric consistency across consecutive video frames as a supervisory signal. The core idea is that a depth prediction model and a camera pose estimation model can be jointly trained to enable view synthesis.

• Core Principle: A point projected from a source image into a target view should have the same appearance. The photometric reprojection error between the synthesized and real target image serves as the training loss.
• Key Components: Requires a differentiable image warping module using spatial transformer networks and often incorporates an explainability mask to handle occlusions and non-Lambertian surfaces.
• Advantage: Enables training on vast, unlabeled video datasets (e.g., YouTube, driving footage), dramatically improving generalization.

Improves depth estimation accuracy and robustness by jointly learning related tasks, forcing the model to develop a more comprehensive understanding of the scene.

• Common Auxiliary Tasks: Semantic segmentation, surface normal estimation, and camera pose estimation. The shared feature representation benefits all tasks.
• Architectural Pattern: A shared backbone encoder (e.g., ResNet, EfficientNet) extracts features, with separate task-specific decoder heads.
• Benefit: Provides implicit regularization, leading to more geometrically coherent depth predictions, especially at object boundaries and in textureless regions.

State-of-the-art models leverage the transformer architecture's ability to model global context, overcoming the limited receptive field of traditional CNNs.

• Vision Transformers (ViTs): Treat the image as a sequence of patches. Models like DPT (Dense Prediction Transformer) use a ViT backbone with a convolutional decoder to produce high-resolution, detail-preserving depth maps.
• Convolutional Vision Transformers (CViTs): Hybrid models that combine convolutional inductive bias for local feature extraction with transformer blocks for global reasoning (e.g., BinsFormer).
• Advantage: Excels at capturing long-range dependencies (e.g., the relationship between a vanishing point and scene scale), leading to superior global scale consistency.

Two fundamental formulations for representing the depth output space.

• Continuous Regression: The model directly predicts a floating-point depth value for each pixel. This is the most direct approach but can be challenging to optimize.
• Ordinal/Discrete Classification (Depth Bins): A more recent, effective approach where depth range is discretized into bins. The model predicts a probability distribution over these bins for each pixel (e.g., AdaBins). The final depth is calculated as a linear combination of bin centers weighted by the probabilities.
• Advantage of Binning: Transforms the problem into classification, which is often easier for networks to learn, and provides useful ordinal constraints that improve edge sharpness.

Emerging methods that leverage massive pre-trained models (e.g., CLIP, DINOv2, diffusion models) to estimate depth without any task-specific training on depth data.

• Principle: Uses the rich, semantically-aware features from foundation models as a prior. For example, Marigold leverages a pre-trained text-to-image diffusion model (Stable Diffusion) and applies denoising diffusion to iteratively refine a depth map from a single image.
• Prompt-Based Methods: Some approaches use natural language prompts (e.g., "a photo of a room") to guide the depth estimation process via a vision-language model.
• Significance: Represents a shift towards general-purpose visual understanding, enabling depth estimation on any image with no fine-tuning, though often at higher computational cost.

A depth map is a 2D image where each pixel's value represents the distance from the camera plane to the corresponding 3D point in the scene. It is the direct output of a monocular depth estimation model.

• Structure: Typically a single-channel grayscale image, where brightness corresponds to distance (brighter = closer).
• Usage: Serves as the primary input for converting 2D pixels into 3D point clouds and is a critical component in RGB-D reconstruction pipelines.
• Example: In autonomous driving, a depth map generated from a front-facing camera is used to estimate the distance to vehicles and pedestrians.

Structure from Motion (SfM) is a classical photogrammetry technique that reconstructs sparse 3D point clouds and estimates camera poses from a collection of unordered 2D images.

• Key Difference from Monocular Depth: SfM uses multiple images to triangulate 3D points, whereas monocular depth estimates from a single image.
• Process: Relies heavily on feature matching and robust estimation algorithms like RANSAC to filter outliers.
• Output: Produces a sparse reconstruction, which is often passed to Multi-View Stereo (MVS) for densification.

Multi-View Stereo (MVS) is a technique that generates dense 3D geometry (a point cloud or mesh) from multiple, calibrated images of a static scene. It builds upon the camera poses estimated by SfM.

• Core Function: For each pixel in a reference image, MVS searches for correspondences in other views to compute depth, creating a dense depth map per view.
• Comparison: Unlike learning-based monocular depth, MVS is a geometric method that requires known camera parameters and multiple overlapping images.
• Challenge: Performance degrades in textureless regions or with imperfect camera calibration.

Visual SLAM (Simultaneous Localization and Mapping) is a real-time process where an agent (e.g., robot, phone) builds a map of its environment and localizes itself within that map using primarily visual input.

• Relationship to Depth: Many Visual SLAM systems incorporate monocular depth estimation as a module to create denser maps without requiring stereo cameras or active depth sensors.
• Key Components: Includes camera pose tracking, local mapping, and loop closure detection.
• Application: Essential for augmented reality (AR) anchoring and robotic navigation in unknown spaces.

Inverse rendering is the process of inferring underlying scene properties—including geometry (often represented as a depth map or mesh), material reflectance, and lighting—from a set of 2D observations.

• Connection: Monocular depth estimation can be seen as a simplified, geometry-focused subset of inverse rendering.
• Advanced Methods: Modern approaches use differentiable rendering to optimize neural representations of scene properties by comparing rendered and real images.
• Goal: To obtain a disentangled, editable 3D scene representation suitable for relighting and object insertion.

A Truncated Signed Distance Function (TSDF) is a volumetric representation used to fuse multiple depth measurements (e.g., from a moving depth sensor or estimated depth maps) into a consistent 3D model.

• Mechanism: Each voxel in a 3D grid stores the signed distance to the nearest surface (negative inside, positive outside), truncated to a fixed range.
• Fusion: As new depth maps are aligned, their distances are averaged into the TSDF volume. A zero-crossing of the TSDF indicates the surface.
• Extraction: The final mesh is extracted using the Marching Cubes algorithm. This is a core technique in RGB-D reconstruction systems like KinectFusion.