Depth Map

The core property of a depth map is that each pixel value represents the distance from the camera's optical center to the corresponding 3D point in the scene. This is typically measured along the camera's Z-axis (optical axis).

• Metric Depth: Values are in real-world units (e.g., meters, millimeters).
• Disparity: Often used in stereo vision, disparity is the inverse of depth and is proportional to the horizontal pixel shift between corresponding points in a stereo pair.
• Normalization: For visualization or processing, raw depth values are often normalized to an 8-bit or 16-bit range, but the original metric scale is crucial for accurate 3D reconstruction.

Depth maps are generated by active sensors, passive algorithms, or learned models.

• Active Sensors: LiDAR uses laser pulses, while Structured Light (e.g., Kinect) projects a known pattern. Time-of-Flight (ToF) sensors measure the round-trip time of emitted light.
• Passive Stereo: Uses two calibrated cameras and stereo matching algorithms to compute a disparity map, which is converted to depth.
• Monocular Estimation: Deep learning models (e.g., MiDaS) predict depth from a single RGB image, but scale is ambiguous without additional cues.
• Fusion (RGB-D): Combines synchronized color and depth data, as from an Intel RealSense camera, for richer scene understanding.

Depth maps are structured as 2D arrays, but their interpretation is inherently 3D.

• Single-Channel Image: Typically a grayscale image where brightness corresponds to distance (brighter = closer, or vice-versa).
• Floating-Point Precision: For accuracy, depth is often stored as 32-bit float values, not integers.
• Missing Data (Holes): Pixels may have invalid values (NaN, 0) where depth sensing failed due to occlusion, reflective surfaces, or sensor limitations. Hole-filling is a common post-processing step.
• Relationship to 3D: A depth map, combined with camera intrinsics, can be back-projected to create a 3D point cloud using the pinhole camera model: (x, y, z) = ( (u - cx) * d / fx, (v - cy) * d / fy, d ) where (u,v) is the pixel, d is its depth value, and (fx, fy, cx, cy) are the camera's focal lengths and principal point.

Depth maps are the primary input for many dense 3D reconstruction pipelines.

• Volumetric Fusion: Multiple aligned depth maps are fused into a global Truncated Signed Distance Function (TSDF) volume, which is then meshed using the Marching Cubes algorithm. This is the core of KinectFusion.
• Dense SLAM: Real-time systems like ElasticFusion or BundleFusion use depth frames for simultaneous mapping and localization.
• Mesh Refinement: Depth maps can be used to refine or sculpt the geometry of an existing 3D mesh.
• View Synthesis & Inpainting: Depth enables advanced image-based rendering techniques, allowing for realistic synthesis of novel viewpoints.

Despite their utility, depth maps have inherent limitations that affect downstream tasks.

• Sensor Noise: Especially at longer ranges, depth measurements can be noisy or inconsistent.
• Occlusions & Shadows: Active sensors cannot see behind objects, creating depth discontinuities and missing data.
• Material Dependence: Transparent, reflective, or absorptive surfaces (e.g., glass, mirrors, black fabric) often return incorrect or no depth data.
• Limited Field of View & Range: Each sensor has minimum/maximum operational distances and an angular field of view.
• Temporal Consistency: For dynamic scenes, aligning depth maps across time requires robust tracking and often leads to motion blur artifacts in the depth channel.

Depth maps are often used in conjunction with or converted into other 3D representations.

• Point Cloud: The direct 3D result of back-projecting a depth map. It is unstructured and can be processed by PointNet-style networks.
• Voxel Grid: A regular 3D grid where depth data can be accumulated. Useful for convolutional 3D networks but memory-intensive.
• Mesh: A surface representation (vertices and faces) extracted from fused depth data via surface reconstruction.
• Normal Map: Can be computed from a depth map by estimating surface gradients. It encodes local surface orientation and is critical for realistic lighting in rendering.
• Semantic/Instance Segmentation: When combined with depth, 2D semantic labels can be lifted to 3D, enabling semantic reconstruction where objects in the 3D model are categorized.

In AR/VR, depth maps enable realistic occlusion and physics-based interactions. They allow virtual objects to appear behind or in front of real-world surfaces and enable gestures to manipulate digital content. Spatial computing architectures use real-time depth sensing for scene understanding and mesh generation, creating persistent AR experiences. This is critical for applications ranging from furniture placement apps to industrial maintenance guides.

Depth maps enable advanced photo effects that were once only possible with professional hardware. Key applications include:

• Portrait Mode (Bokeh): Artificially blurring the background based on subject distance.
• Refocusing: Allowing users to change the focal point of an image after it's taken.
• 3D Photos: Creating stereoscopic images or parallax effects for social media. These features are now standard in modern smartphones, powered by monocular depth estimation or multi-camera stereo matching.

In film and game production, depth maps (often called Z-depths) are used for compositing and rendering efficiency.

• Compositing: Seamlessly integrating CGI elements into live-action footage by ensuring correct layering and atmospheric perspective.
• Depth-Based Effects: Simulating fog, depth-of-field, and other volumetric effects that rely on distance.
• Motion Graphics: Creating parallax scrolling and 2.5D effects from a single image. The depth map acts as a 3D proxy for post-processing.

When combined with RGB data, depth maps significantly improve the accuracy of identifying and separating objects in a scene. The added spatial context helps disambiguate objects that are similar in color but at different distances. This enhanced semantic segmentation and instance segmentation is vital for:

• Autonomous Vehicles: Distinguishing a pedestrian from a poster on a wall.
• Robotic Manipulation: Enabling a robot arm to segment and grasp a specific item from a cluttered bin.
• Geospatial Analysis: Classifying terrain and structures from aerial imagery.

A point cloud is a set of discrete data points in a 3D coordinate system, representing the external surface of an object or scene. It is a direct 3D representation derived from depth maps, LiDAR scans, or photogrammetry.

• Generation: Created by back-projecting depth map pixels into 3D space using camera parameters.
• Properties: Contains XYZ coordinates and often color (RGB) or intensity values for each point.
• Applications: Used as input for surface reconstruction, robot navigation, and as a lightweight scene representation.

A Truncated Signed Distance Function (TSDF) is a volumetric representation that fuses multiple depth maps into a globally consistent 3D model. For each voxel in a 3D grid, it stores the signed distance to the nearest surface.

• Core Function: Distances are positive in front of the surface, negative behind, and truncated to a fixed range for efficiency.
• Fusion: New depth frames are integrated via weighted averaging, smoothing noise and filling holes.
• Output: The zero-crossing of the TSDF defines the reconstructed surface, which can be extracted using algorithms like Marching Cubes. This is the foundation of real-time 3D reconstruction systems like KinectFusion.

Stereo matching is the core algorithm for generating depth maps from a pair of rectified stereo cameras. It produces a disparity map, where each pixel value represents the horizontal shift (in pixels) between corresponding points in the left and right images.

• Process: For each pixel in the reference image, the algorithm searches along the same row in the target image to find the best match.
• Conversion: Depth is calculated from disparity using the formula: Depth = (Focal Length * Baseline) / Disparity.
• Challenges: Includes handling occlusions, textureless regions, and repetitive patterns. Modern approaches use deep learning for robust matching.

Monocular depth estimation is the task of predicting a depth map from a single 2D RGB image, using learned priors about the 3D world. It is an ill-posed problem solved with deep learning.

• Approach: Uses convolutional neural networks (CNNs) or vision transformers trained on datasets with ground-truth depth (e.g., from LiDAR).
• Output: Produces relative or metric depth, indicating the scale of objects relative to each other or in absolute units.
• Use Case: Critical for applications where stereo or active depth sensors are unavailable, such as in standard smartphone photography or video analysis.

A surface normal is a vector that is perpendicular to a surface at a given point. In the context of depth maps, normals describe local surface orientation and are crucial for lighting, reconstruction, and robotics.

• Calculation: Computed from a depth map by analyzing the local neighborhood of pixels to estimate the tangent plane.
• Representation: Often stored in a normal map, a 2D image where the RGB channels encode the (X, Y, Z) components of the normal vector.
• Applications: Used in Phong shading for realistic rendering, in TSDF fusion to improve integration quality, and for obstacle detection in robotics (e.g., identifying traversable slopes).

An RGB-D image is a synchronized data frame containing both a standard color (RGB) image and a per-pixel depth (D) map. It is the primary data format for many 3D perception systems.

• Sensors: Captured by devices like the Microsoft Kinect, Intel RealSense, and Apple LiDAR Scanner.
• Alignment: The depth map is intrinsically aligned with the color image, meaning each depth pixel corresponds directly to an RGB pixel.
• Utility: Provides dense, textured 3D information in a single frame. This is the foundational input for RGB-D SLAM, 3D object recognition, and interactive AR applications.