Depth Map

The core data stored in a depth map is a metric distance for each pixel. This value is typically measured in real-world units (e.g., meters, millimeters) along the camera's optical axis (z-axis).

• Absolute vs. Relative Depth: Systems like LiDAR or structured light sensors produce absolute metric depth. Stereo vision and monocular depth estimation often produce relative depth, which must be scaled using known reference distances.
• Storage Formats: Depth values are commonly stored as 16-bit unsigned integers or 32-bit floating-point numbers in a single-channel image, separate from the RGB color channels.

Depth maps are generated through active sensing, passive vision, or learned inference, each with distinct trade-offs in accuracy, range, and environment compatibility.

• Active Sensing (LiDAR, Structured Light): Projects infrared patterns or laser pulses to measure time-of-flight or pattern deformation. Provides high accuracy but can be affected by sunlight and specular surfaces. Used in Apple's TrueDepth camera and autonomous vehicle LiDAR.
• Passive Stereo Vision: Calculates depth by finding correspondences between two or more camera views (triangulation). Computationally intensive but requires no active emission. Found in stereo camera rigs.
• Monocular Depth Estimation: Uses a trained convolutional neural network (CNN) to predict depth from a single 2D image. While not metrically precise without scaling, it's invaluable for applications where only a single camera is available.

Depth maps are the primary input for converting 2D imagery into explicit 3D geometry. They bridge the gap between image space and 3D world coordinates.

• Point Cloud Generation: Each pixel (u, v) with depth d is back-projected into 3D space using the camera's intrinsic matrix, creating a 3D point cloud.
• Surface Reconstruction: Multiple aligned depth maps (from different viewpoints) are fused using algorithms like KinectFusion or TSDF (Truncated Signed Distance Function) integration to create a watertight polygonal mesh.
• Dense vs. Sparse: Dense reconstruction uses every pixel from a depth map. Sparse reconstruction, like classic Structure-from-Motion, uses only tracked feature points.

Depth maps enable machines to perceive and interact with the physical 3D world, forming the backbone of spatial computing.

• Augmented Reality Occlusion: Virtual objects are correctly obscured by real-world geometry, enhancing realism. ARKit and ARCore use depth for this.
• Robotic Navigation & Manipulation: Provides the 3D obstacle map essential for path planning (e.g., in robotic vacuum cleaners or warehouse AMRs) and for calculating grasp poses.
• 3D Scene Understanding: Combined with semantic segmentation, depth maps allow for reasoning about object volumes, spatial relationships, and scene layout.

Real-world depth data is imperfect. Understanding its failure modes is critical for building robust systems.

• Sensor Noise: Exhibits as speckle or flickering in the depth values. Multi-path interference occurs when signals bounce off multiple surfaces before returning.
• Missing Data (Holes): Caused by specular reflections, absorbent materials (black surfaces), transparent objects, or distances beyond the sensor's operational range.
• Motion Artifacts: In frame-by-frame systems, movement during capture causes misalignment between the RGB and depth images, known as motion blur in the depth domain.

Raw depth maps are often processed to improve quality and fused with other data for more complete spatial understanding.

• Filtering: Bilateral filters and median filters reduce noise while preserving edges. Hole-filling algorithms interpolate missing regions.
• Temporal Fusion: Averaging depth over multiple frames reduces noise and fills transient holes, at the cost of latency.
• Sensor Fusion with VIO: Depth maps are fused with Visual-Inertial Odometry (VIO) pose estimates to build globally consistent 3D maps. This is a core component of Visual SLAM systems like ORB-SLAM3.

A point cloud is a set of discrete data points in a 3D coordinate system, representing the external surfaces of objects or environments. It is a direct, unorganized output from depth sensors like LiDAR or stereo cameras.

• Relationship to Depth Maps: A depth map can be directly converted into a point cloud by projecting each pixel's 2D coordinates into 3D space using its depth value and the camera's intrinsic parameters.
• Primary Use: Serves as the raw geometric input for surface reconstruction, SLAM, and collision detection.
• Example: Autonomous vehicles use LiDAR-generated point clouds to perceive the precise 3D shape of nearby vehicles and obstacles.

Simultaneous Localization and Mapping (SLAM) is the computational problem of constructing a map of an unknown environment while simultaneously tracking an agent's location within it.

• Role of Depth: Depth maps (from RGB-D cameras, stereo, or inferred via monocular depth estimation) provide the 3D measurements essential for building a metric map and estimating the agent's 6DoF pose.
• Key Algorithms: Includes Visual SLAM (vSLAM) and LiDAR SLAM. Systems like ORB-SLAM3 use feature tracking and bundle adjustment with depth data.
• Application: Foundational for robot navigation, augmented reality world tracking, and autonomous drone flight.

Surface reconstruction is the process of creating a continuous, watertight polygonal mesh or other explicit surface model from a set of unorganized 3D points, such as those from a point cloud.

• Input Data: Often begins with a depth map converted to a point cloud.
• Core Techniques:
  • Poisson Surface Reconstruction: Creates a smooth surface by solving an Poisson equation.
  • Marching Cubes: Extracts a polygonal mesh from a volumetric scalar field, like a Truncated Signed Distance Function (TSDF).
• Output: A mesh usable for rendering, 3D printing, or physics simulations in digital twins.

A Signed Distance Function (SDF) is an implicit neural or volumetric representation where, for any 3D coordinate, the value represents the shortest distance to the surface of an object, with sign indicating inside (negative) or outside (positive).

• Contrast with Depth Maps: An SDF is a continuous 3D field, whereas a depth map is a 2D projection of distance from a single viewpoint.
• Usage in NeRF/3D AI: Modern neural scene representations like NeuS and Instant-NGP use SDFs or similar fields to model geometry with high fidelity, enabling high-quality surface reconstruction from multi-view images.
• Advantage: Provides a clean, differentiable representation ideal for optimization and rendering.

Visual-Inertial Odometry (VIO) is a sensor fusion technique that combines visual data from a camera with inertial data from an IMU (gyroscope, accelerometer) to estimate the device's 6-degree-of-freedom pose and trajectory.

• Depth Integration: While often monocular, VIO systems benefit greatly from dense depth maps (e.g., from an RGB-D camera) to create a metric-scale map and improve tracking robustness, especially during rapid motion or visual texture loss.
• Mechanism: The IMU provides high-frequency motion estimates, which are corrected and scaled by lower-frequency, drift-free visual observations that often rely on triangulating features with estimated depth.
• Industry Standard: The core tracking technology in mobile AR platforms like ARKit and ARCore.

Semantic segmentation is a pixel-level classification task that assigns a categorical label (e.g., 'road', 'person', 'building') to every pixel in an image.

• Fusion with Depth: Combining a depth map with a semantic segmentation map creates a semantic point cloud or 3D semantic map. This is crucial for scene understanding, allowing systems to reason not just about geometry but also object identity and function.
• Application in Autonomy: An autonomous vehicle uses semantic segmentation on camera images, fused with LiDAR depth, to understand that a distant red cluster is a traffic light and a nearby vertical surface is a pedestrian.
• Advanced Models: Architectures like Mask R-CNN and Segment Anything Model (SAM) provide the 2D masks that are lifted into 3D using depth.