Monocular Depth Estimation

The core challenge of monocular depth estimation is the inverse projection problem: a single 2D image corresponds to an infinite number of possible 3D scenes. A key manifestation is scale ambiguity—the absolute distance to objects cannot be determined from a single image alone without a known reference. For example, a small, nearby object can project the same image as a large, distant one. Models overcome this by learning statistical priors from training data, such as typical object sizes, perspective cues, and scene layouts.

Since direct geometric measurement is impossible, algorithms must learn to interpret a hierarchy of monocular depth cues:

• Geometric Cues: Linear perspective, vanishing points, and relative object size.
• Photometric Cues: Shading, shadows, and texture gradients (texture appears denser with distance).
• Semantic Priors: Learned associations (e.g., a 'car' is typically ~1.5m tall, the 'sky' is distant).
• Motion Priors (in video): Parallax from ego-motion provides strong cues for relative depth. Modern deep learning models implicitly encode these priors from massive datasets.

The standard output is a depth map (or disparity map)—a per-pixel array where each value represents the distance from the camera plane to the corresponding scene point. It can be represented as:

• Metric Depth: Absolute distance in real-world units (e.g., meters). Requires scale-aware training.
• Relative/Ordinal Depth: Indicates which points are closer/farther than others, without absolute scale. Often sufficient for tasks like obstacle avoidance.
• Disparity: The inverse of depth (1/distance), which is proportional to the shift needed for stereo matching and is linear in image coordinates, making it easier for networks to regress.

Model architectures have evolved to tackle this dense prediction task:

• Encoder-Decoder CNNs: Early models used a U-Net-like structure. An encoder (e.g., ResNet) extracts hierarchical features, and a decoder upsamples to full resolution, often using skip connections to preserve fine details.
• Vision Transformers (ViTs): Models like DPT (Dense Prediction Transformers) treat image patches as sequences, using self-attention to capture global context crucial for resolving ambiguities (e.g., relating a distant building to the horizon).
• Multi-Task & Multi-Modal Models: Jointly predicting depth alongside semantic segmentation or using vision-language models (VLMs) like CLIP for improved semantic priors.

Methods are defined by their training signal:

• Fully Supervised: Trained on ground-truth depth from LiDAR or RGB-D sensors (e.g., KITTI, NYU Depth V2). Uses losses like Scale-Invariant Log Loss to focus on relative accuracy.
• Self-Supervised / Unsupervised: Uses stereo image pairs or monocular video sequences. The core idea is photometric consistency: the warped left image should match the right image. A neural network predicts depth to enable this warping, and the photometric error becomes the training signal. This avoids costly ground-truth data.
• Semi-Supervised & Domain Adaptation: Combines limited real-world labels with abundant synthetic data from engines like CARLA, using techniques to bridge the sim-to-real domain gap.

Monocular depth is critical for robots due to its low SWaP (Size, Weight, and Power) requirements:

• Obstacle Avoidance & Navigation: Provides essential 3D geometry for path planning in visual SLAM (vSLAM) and visual odometry pipelines.
• Manipulation & Grasping: Estimates distance to objects for visual servoing and calculating grasp points.
• Scene Understanding: Combined with semantic segmentation, it enables 3D semantic scene reconstruction.
• Augmented Reality (AR): Anchors virtual objects convincingly in the real world by understanding scene geometry. Its primary limitation remains accuracy at long range compared to active sensors like LiDAR.

The process of estimating the ego-motion (rotation and translation) of a camera by analyzing the sequential changes in a stream of images. Unlike monocular depth estimation, which provides a static 3D snapshot, VO focuses on tracking movement over time.

• Core Mechanism: Typically involves feature tracking or direct methods to estimate the camera's pose between consecutive frames.
• Key Challenge: Scale ambiguity is a fundamental issue in pure monocular VO, as depth is only estimated up to an unknown scale factor.
• Application: A critical component for robot navigation and a precursor to more complex systems like Visual SLAM.

Visual Simultaneous Localization and Mapping is a computational technique that enables an agent to construct a map of an unknown environment and simultaneously determine its own location within that map using only visual input.

• Relation to Depth: Monocular depth estimation is often an implicit or explicit component, helping to build a sparse or dense 3D map from 2D images.
• Key Difference: SLAM is a temporal and global optimization problem, maintaining consistency over a long trajectory, whereas single-image depth estimation is a per-frame task.
• Common Frameworks: ORB-SLAM, DSO (Direct Sparse Odometry).

A classical computer vision technique that calculates depth by finding correspondences between pixels in two synchronized images captured from cameras with a known baseline (separation).

• Core Mechanism: Solves the correspondence problem via algorithms like Semi-Global Matching (SGM) to compute a disparity map, which is inversely proportional to depth.
• Contrast with Monocular: Provides metric scale depth by design, as the baseline is a known physical measurement. However, it requires precise calibration and multiple cameras.
• Modern Approach: Often enhanced with deep learning for improved accuracy in textureless regions.

The pattern of apparent motion of image objects, surfaces, and edges between consecutive video frames, caused by the relative movement between the camera and the scene.

• Relation to Depth: Optical flow is a 2D motion field. When combined with known camera motion (egomotion), it provides strong cues for scene geometry and relative depth (motion parallax).
• Key Distinction: Optical flow estimates pixel-wise 2D displacement, not absolute 3D distance. It is a fundamental input for many self-supervised monocular depth and VO training paradigms.
• Application: Used for motion segmentation, video compression, and as a supervisory signal in self-supervised learning.

The computer vision and graphics task of generating a photorealistic image of a scene from a camera viewpoint that was not present in the original set of input images.

• Deep Connection: High-quality monocular depth estimation, often coupled with camera pose, is a critical intermediate representation for view synthesis. Techniques like Neural Radiance Fields (NeRF) implicitly learn a continuous 3D representation that includes geometry (akin to depth) and appearance.
• Application: Essential for creating immersive experiences, digital twins, and data augmentation for robotics training in simulation.

A dense prediction task that assigns a categorical label (e.g., 'road', 'person', 'building') to every pixel in an image.

• Synergy with Depth: Depth-aware segmentation and semantically-guided depth estimation are active research areas. Knowing an object's class (e.g., 'sky' is far, 'car' has bounded size) provides strong priors for depth prediction, and vice versa.
• Joint Models: Many modern architectures are multi-task networks that predict depth, segmentation, and surface normals from a single image, sharing a common encoder for efficiency.
• Robotics Impact: Crucial for a robot to understand not just where things are (depth), but what they are (semantics) for safe navigation and manipulation.