Monocular Depth Estimation

Classical methods rely on monocular cues present in a single image to infer depth. These are heuristics derived from human visual perception and projective geometry.

Key cues include:

• Relative Size & Scale: Known object sizes provide distance references.
• Linear Perspective: Parallel lines converge at a vanishing point.
• Texture Gradient: Surface textures become denser with distance.
• Occlusion: Closer objects block the view of farther ones.
• Shading & Lighting: Variations in light and shadow reveal surface orientation.

While interpretable, these methods are highly scene-dependent and lack the robustness of data-driven approaches for general environments.

This is the dominant paradigm, where a convolutional neural network (CNN) is trained on large datasets of images paired with ground-truth depth maps from sensors like LiDAR or RGB-D cameras.

Architectural Hallmarks:

• Encoder-Decoder Networks: The encoder (e.g., ResNet) extracts hierarchical features; the decoder upsamples to produce a full-resolution depth map.
• Skip Connections: Direct links between encoder and decoder layers (as in U-Net) preserve fine spatial details.
• Multi-Scale Prediction: Predicting depth at multiple resolutions improves both local accuracy and global structure.

Loss Functions combine per-pixel depth error (L1/L2) with gradient-based terms to enforce smoothness and preserve edges. Trained models like MiDaS and DPT demonstrate strong generalization across diverse scenes.

These methods train networks without ground-truth depth by leveraging geometric constraints from video sequences or stereo image pairs.

The core principle is view synthesis: the network learns depth and camera ego-motion by trying to reconstruct one image from another.

Key Components:

• Depth CNN: Predicts a depth map for a target image.
• Pose CNN: Predicts the relative camera transformation between frames.
• Differentiable Warping: Uses predicted depth and pose to warp a source image into the target view via a spatial transformer network.

The network is trained to minimize the photometric reprojection error (e.g., L1 difference) between the synthesized and real target image. Masking is used to ignore occluded regions and non-rigid motion. This approach is highly scalable but can struggle in low-texture or non-Lambertian regions.

Inspired by success in NLP and vision, Vision Transformers (ViTs) and Dense Prediction Transformers (DPT) have become state-of-the-art for monocular depth.

Key Advantages:

• Global Context: Self-attention mechanisms model long-range dependencies across the entire image, crucial for resolving ambiguous regions (e.g., large uniform surfaces).
• Multi-Head Attention: Allows the model to focus on different types of visual relationships (edges, textures, semantics) simultaneously.

Models like DPT-Hybrid use a ViT backbone pre-trained on large-scale image classification, fine-tuned for depth. The transformer encoder creates a set of feature tokens, which a convolutional decoder reassembles into a high-resolution depth map. This architecture excels at capturing the structural coherence of complex scenes.

Depth estimation is often learned jointly with related vision tasks, leveraging shared visual features for mutual improvement.

Common Co-Tasks:

• Semantic Segmentation: Knowing an object's class (e.g., 'sky' is far, 'car' has bounded size) provides strong priors for depth.
• Surface Normals: Normals describe local orientation, which is geometrically linked to depth gradients.
• Edge Detection: Depth discontinuities often align with object boundaries.

Implementation: Networks have a shared encoder with multiple task-specific decoder heads. The training loss is a weighted sum of individual task losses. This approach leads to more geometrically consistent and semantically aware depth predictions, as the model is forced to learn a richer, more generalizable representation.

Moving beyond discrete pixel grids, these methods represent depth or the full 3D scene as a continuous function parameterized by a neural network.

Core Idea: A multilayer perceptron (MLP) learns a mapping from image coordinates (and optionally a latent code) to a depth value.

Benefits:

• Resolution-Free: The function can be queried at any spatial coordinate, enabling arbitrary output resolution.
• Memory Efficiency: The scene is compressed into network weights, not a large dense map.
• Smoothness Priors: The network's inherent bias towards low-frequency functions acts as a natural smoothness regularizer.

This paradigm is closely related to Neural Radiance Fields (NeRF) for view synthesis. For monocular depth, it allows elegant integration of uncertainty estimation and can generate highly detailed, continuous depth surfaces.

A photogrammetry technique that reconstructs the 3D structure of a scene and the camera poses from a collection of unordered 2D images. It works by detecting distinctive keypoints, matching them across images, and solving for geometry and motion through bundle adjustment. Unlike monocular depth estimation from a single frame, SfM is an offline, multi-view process that produces a sparse 3D point cloud.

A computer vision technique that generates a dense 3D reconstruction (e.g., a point cloud or mesh) from multiple calibrated images with known camera positions. MVS typically follows SfM and uses photometric consistency across views to estimate depth for every pixel. It provides geometrically accurate results but requires precise camera calibration and multiple viewpoints, whereas monocular depth estimation aims for a similar output from a single image.

The process of converting a sparse depth map (e.g., from a LiDAR sensor with ~1% pixel coverage) into a dense, pixel-aligned depth map. It uses a corresponding RGB image as guidance to infer the missing depth values via deep learning or optimization. This task is closely related to monocular depth estimation but is semi-supervised, as it starts with some ground-truth depth measurements to guide the prediction.

The simultaneous capture of color (RGB) and per-pixel depth (D) information. This is typically achieved with integrated sensors like Time-of-Flight (ToF) cameras or structured light systems (e.g., Microsoft Kinect). RGB-D data provides direct ground truth for training monocular depth estimation models and is fundamental for applications in robotics and augmented reality where real-time depth is required.

The process of estimating a camera's ego-motion (position and orientation) by analyzing the sequence of images it captures. Monocular VO uses only a single camera and must estimate both motion and scene scale, often relying on techniques like feature matching and bundle adjustment. Accurate depth estimation, whether from monocular networks or other sensors, is critical for scaling the motion estimates correctly.

A method of encoding a 3D shape or scene using a neural network (typically a multilayer perceptron) that maps 3D spatial coordinates to properties like occupancy, signed distance, or color. Examples include Neural Radiance Fields (NeRF). These representations offer a continuous, memory-efficient alternative to explicit 3D data (like point clouds). Monocular depth estimation can be a first step towards building such an implicit model from a single image.