Depth Completion

The core task is to infer missing depth values for every pixel in an image, starting from a sparse set of known measurements. This is inherently an ill-posed problem, as many plausible 3D geometries can produce the same 2D image. Algorithms must leverage photometric consistency, geometric priors, and semantic context to resolve ambiguities. For example, a region identified as 'sky' should be assigned a very large depth, while a 'car' has bounded dimensions.

• Input: A sparse depth map (e.g., from LiDAR) aligned with an RGB image.
• Output: A dense, pixel-aligned depth map.
• Challenge: The sparsity pattern is irregular and non-uniform, unlike a regular grid of missing pixels.

Depth completion is a quintessential sensor fusion task, most commonly fusing LiDAR with camera data. The LiDAR provides accurate but sparse geometric ground truth, while the camera provides dense photometric and semantic cues to guide interpolation.

• LiDAR-Camera Calibration: Precise spatial and temporal alignment between sensors is a prerequisite. Errors here directly propagate to the output.
• Modality Strengths: LiDAR gives precise range but poor texture/color. Cameras give rich texture but no direct range.
• Fusion Level: Fusion can happen at the early (input concatenation), mid (feature-level), or late (output refinement) stage in a neural network.

Modern depth completion is dominated by deep convolutional neural networks (CNNs) and vision transformers, which learn to correlate image features with depth patterns.

• Encoder-Decoder Networks: Common architecture where an encoder extracts multi-scale features from the RGB and sparse depth inputs, and a decoder upsamples to produce the dense map.
• Sparse Convolutions: Specialized layers that operate only on active (non-zero) input locations, dramatically improving efficiency for sparse data.
• Guided Filtering: Post-processing or integrated modules that use the RGB image's edges to sharpen depth discontinuities, preventing 'bleeding' of depth across object boundaries.

A high-quality completed depth map must exhibit specific properties to be useful for robotics:

• Edge-Awareness: Depth discontinuities must align precisely with object boundaries in the RGB image. Blurry or smoothed edges cause errors in obstacle detection.
• Metric Accuracy: The predicted depth values must be metrically accurate, not just relatively correct. This is crucial for path planning and manipulation.
• Temporal Consistency: For video streams, depth maps should be stable frame-to-frame to avoid jitter in the perceived 3D world.
• Robustness to Sparsity: Performance should degrade gracefully as input sparsity increases (e.g., with longer-range LiDAR returns).

Progress is driven by large-scale, real-world datasets. Key benchmarks include:

• KITTI Depth Completion: The seminal benchmark using Velodyne 64-beam LiDAR and stereo cameras from a driving platform. It defines standard train/test splits and evaluation metrics (RMSE, MAE).
• NYU Depth V2: An indoor dataset using a Microsoft Kinect (RGB-D sensor), often used for denser, indoor completion tasks.
• Waymo Open Dataset: Provides higher-resolution LiDAR and cameras, presenting challenges with greater sparsity and diversity.
• Evaluation Metrics: Root Mean Square Error (RMSE) is the primary metric, penalizing large errors. Mean Absolute Error (MAE) and Accuracy under thresholds (δ) are also standard.

Dense depth is a foundational layer for numerous embodied AI functions:

• 3D Object Detection: Converts 2D image-based detections into oriented 3D bounding boxes.
• Occupancy Grid Mapping: Creates a robot-centric map for navigation by projecting dense depth into voxels.
• Motion Planning: Provides the geometric scene understanding necessary for calculating collision-free paths.
• Augmented Reality: Enables realistic occlusion and physics-based interaction between virtual and real objects.
• Sim-to-Real Transfer: Completed depth can provide a more simulation-like perceptual input, bridging the reality gap for policies trained in sim.

The task of predicting a dense depth map from a single RGB image without any explicit depth sensor input. Unlike depth completion, which starts with sparse sensor data, this is a purely vision-based inference problem.

• Key Challenge: It's an ill-posed problem (infinite 3D scenes can project to the same 2D image).
• Approach: Uses deep learning models (e.g., convolutional neural networks) to learn cues like perspective, texture, and object size from large datasets.
• Application: Foundational for augmented reality, basic robot navigation, and generating pseudo-LiDAR data for autonomous driving perception systems.

The simultaneous capture of color (RGB) and per-pixel depth (D) information, typically from a single integrated sensor like a Microsoft Kinect or Intel RealSense camera.

• Sensor Types: Includes Structured Light and Time-of-Flight (ToF) technologies.
• Output: Produces an aligned dense depth map by design, making depth completion unnecessary for its raw data.
• Contrast with LiDAR: RGB-D sensors are lower-cost and shorter-range but provide denser, pixel-aligned depth ideal for indoor robotics and human-computer interaction.

A sensor fusion technique that combines the complementary strengths of cameras and LiDAR. The camera provides high-resolution texture and semantic information, while LiDAR provides precise, long-range geometric measurements.

• Core Problem: The data is inherently misaligned—LiDAR points are sparse and in a 3D coordinate system, while camera pixels are dense and 2D.
• Role of Depth Completion: Often a critical intermediate step to project sparse LiDAR points onto the image plane and generate a dense, context-aware depth map that leverages color cues.
• Application: Essential for robust perception in autonomous vehicles and advanced robotics.

The task of assigning a semantic class label (e.g., 'car', 'pedestrian', 'building') to each element within a 3D representation, such as a point cloud or voxel grid.

• Relationship to Depth Completion: A completed dense depth map can be back-projected to create a denser 3D point cloud, which serves as superior input for 3D segmentation networks compared to raw sparse LiDAR.
• Joint Learning: State-of-the-art systems often perform depth completion and semantic segmentation jointly in a multi-task network, where each task informs the other for more accurate results.

The process of creating a continuous 2D manifold surface (typically a triangle mesh) from a set of discrete 3D data points, such as a point cloud.

• Input Dependency: The quality of reconstruction is directly dependent on the density and accuracy of the input 3D points.
• Depth Completion's Role: By converting a sparse LiDAR scan into a dense depth map and then a dense point cloud, depth completion provides a far superior starting point for algorithms like Poisson reconstruction or Marching Cubes.
• Application: Critical for creating digital twins, detailed 3D models for inspection, and high-fidelity simulation environments.

A method of encoding a 3D shape or scene using a neural network (e.g., a multi-layer perceptron) that maps 3D spatial coordinates to properties like occupancy or signed distance.

• Contrast with Explicit 3D: Represents geometry as a continuous function rather than discrete points, voxels, or meshes.
• Synergy with Depth Completion: Sparse depth measurements (e.g., from LiDAR) can be used as sparse supervision signals to train a neural implicit model, guiding it to reconstruct the full continuous surface. This is an emerging, learning-based alternative to traditional depth completion pipelines.