Depth Estimation

A classical geometry-based method that calculates depth by finding corresponding pixels in two or more images taken from slightly different viewpoints (a stereo pair).

• Core Principle: Uses triangulation. The displacement of a point between the left and right images, known as disparity, is inversely proportional to its depth.
• Process: Requires epipolar geometry calibration and a computationally intensive correspondence search (e.g., using block matching or semi-global matching).
• Limitations: Struggles with textureless regions, repetitive patterns, and occlusions. Performance depends on a known, fixed baseline (distance between cameras).

A photogrammetry technique that reconstructs sparse 3D point clouds and estimates camera poses from a collection of unordered 2D images of a static scene.

• Core Principle: Solves for 3D structure and camera motion simultaneously by establishing feature correspondences across multiple views and optimizing via bundle adjustment.
• Output: Produces a sparse reconstruction, meaning depth is estimated only for distinctive feature points (e.g., SIFT, ORB).
• Applications: Foundational for creating 3D models from photo collections, mapping, and as a preprocessing step for Multi-View Stereo (MVS).

A deep learning approach that predicts a dense depth map from a single RGB image, learning cues like perspective, object size, texture, and occlusion from vast datasets.

• Core Principle: Treats depth estimation as a supervised regression or classification problem. Models learn a direct mapping from image pixels to depth values.
• Architectures: Typically uses encoder-decoder CNNs (e.g., MiDaS) or vision transformers. Training requires large datasets with ground truth depth (e.g., NYU Depth V2, KITTI).
• Advantage/Challenge: Extremely versatile as it requires only a standard camera, but predictions are inherently scale-ambiguous and rely on learned statistical priors about the world.

An extension of SfM that produces a dense 3D reconstruction (depth map or point cloud) from multiple calibrated images with known camera poses.

• Core Principle: For each pixel in a reference image, it searches along epipolar lines in neighboring source images to find the best-matching patch, then triangulates the 3D position.
• Methods: Ranges from classical patch-based algorithms to modern deep learning MVS networks (e.g., MVSNet) that build cost volumes in 3D space and regularize them with 3D CNNs.
• Output: Creates dense depth maps per view, which can be fused into a complete point cloud or mesh.

Direct measurement techniques that project energy into a scene and measure its return to compute depth, providing high-precision ground truth data.

• LiDAR: Emits laser pulses and measures the time-of-flight (ToF) to create a point cloud. Provides long-range, accurate measurements but can be sparse and expensive.
• Structured Light (e.g., Kinect v1): Projects a known pattern (e.g., infrared dots) onto a scene. Depth is calculated by analyzing the distortion of the pattern, enabling dense, real-time depth at short range.
• Role: Often used as the ground truth source for training and evaluating learning-based monocular or stereo systems.

Training paradigms that learn depth estimation without requiring ground truth depth labels, using photometric consistency as the primary supervisory signal.

• Core Principle: A network predicts depth for a target image and the camera's ego-motion between frames. It then reprojects a neighboring source image into the target view using this depth and pose. The difference between the real target image and the reprojected one (photometric loss) trains the network.
• Requirements: A temporal sequence of images (video) from a moving monocular or stereo camera.
• Benefit: Enables training on vast, unlabeled video datasets, overcoming the scarcity of expensive depth sensor data.

LiDAR is an active remote sensing method that uses pulsed laser light to measure distances, generating highly accurate 3D point clouds. It provides ground-truth depth data for training and validating monocular depth estimation models. Unlike passive cameras, LiDAR operates effectively in low-light conditions but can be affected by weather and is typically more expensive.

• Principle: Measures the time-of-flight of laser pulses.
• Output: Dense set of 3D coordinates (x, y, z) and often intensity.
• Primary Use: Autonomous vehicles, high-precision surveying, and robotics.

A point cloud is a set of data points in a 3D coordinate system, representing the external surface of objects or environments. It is the raw output of depth sensors like LiDAR and RGB-D cameras. Point clouds are unordered and irregular, posing challenges for neural network processing, which led to architectures like PointNet.

• Key Characteristics: Sparse, unstructured, and metric-scale accurate.
• Processing: Requires specialized deep learning models or conversion to voxel grids or meshes.
• Applications: 3D modeling, autonomous navigation, and digital twins.

Stereo vision estimates depth by finding correspondences between two images taken from horizontally offset cameras. The core concept is disparity—the horizontal pixel shift of a point between the left and right images. Depth is inversely proportional to disparity: depth = (baseline * focal_length) / disparity.

• Passive Method: Relies on ambient lighting and texture for matching.
• Algorithm Steps: Rectification, stereo matching (e.g., using Semi-Global Matching), disparity-to-depth conversion.
• Limitation: Struggles with textureless regions, repetitive patterns, and occlusions.

An RGB-D camera is a sensor that captures synchronized RGB color images and per-pixel Depth maps. Common technologies include structured light (e.g., early Microsoft Kinect) and time-of-flight (e.g., Intel RealSense). These sensors provide direct, real-time depth estimates at a lower cost than LiDAR but with shorter range and potential interference from sunlight.

• Structured Light: Projects a known infrared pattern and analyzes its deformation.
• Time-of-Flight: Measures the phase shift or direct time delay of reflected light.
• Primary Datasets: NYU Depth V2, SUN RGB-D, ScanNet.

Monocular depth estimation is the task of predicting a depth map from a single 2D image. It is an inherently ill-posed problem (infinite 3D scenes can project to the same 2D image) solved using deep learning models that learn statistical priors about the world. Approaches are categorized as:

• Supervised: Trained on datasets with ground-truth depth from LiDAR or RGB-D cameras.
• Unsupervised/Self-supervised: Uses geometric constraints from stereo pairs or video sequences (e.g., visual odometry) as training signal, without explicit depth labels.

3D object detection locates and classifies objects in 3D space, predicting their oriented 3D bounding boxes. It relies heavily on accurate depth perception. Methods are sensor-specific:

• LiDAR-based: Operates directly on point clouds (e.g., PointPillars, SECOND).
• Camera-based: Uses monocular or stereo depth estimation to lift 2D image features into 3D (e.g., FCOS3D, DETR3D).
• Multi-sensor Fusion: Combines camera RGB features with LiDAR point features for improved accuracy.
• Key Outputs: 3D center (x, y, z), dimensions (length, width, height), and yaw rotation.