Truncated Signed Distance Function (TSDF)

The foundation of a TSDF is a Signed Distance Function. For any point in 3D space, it stores the shortest distance to the nearest surface. The sign indicates whether the point is inside (negative distance) or outside (positive distance) the object. The zero-level set of this function defines the reconstructed surface.

To make the representation practical and memory-efficient, distances are truncated to a fixed range [-μ, +μ] around the surface. This creates a narrow band of active voxels.

• Why it's critical: It limits updates to the region near the surface, ignoring empty and far-occupied space.
• Impact: Enables the use of a fixed-size voxel grid and guarantees the function is Lipschitz continuous, which aids numerical stability during fusion and mesh extraction.

A TSDF fuses multiple, noisy depth maps from different viewpoints into a single, de-noised model. This is done via a running weighted average in each voxel.

• Process: For each new depth frame, a TSDF slice is projected into the volume. Voxel values (distance D) and weights (W) are updated: D_new = (W_old * D_old + w * d) / (W_old + w).
• Result: Redundant observations average out sensor noise, while occluded areas remain uncertain (low weight).

The continuous TSDF is discretized into a 3D voxel grid. Each voxel stores two values:

1. TSDF Value: The truncated signed distance to the surface.
2. Weight: A confidence measure (often based on sensor noise model or viewing angle).

• Memory Challenge: A dense grid is O(n³). Solutions include hashing (e.g., KinectFusion) or hierarchical structures (octrees) to allocate memory only near surfaces.

To render or mesh the TSDF, the implicit surface (where distance=0) must be extracted. This is done efficiently via ray casting.

• Method: For a desired camera view, rays are cast through the volume. The zero-crossing along each ray is found by interpolating the TSDF values of sampled voxels.
• Output: This generates a depth map and normal map for the reconstructed surface from any novel viewpoint, enabling real-time visualization.

The weighted averaging scheme provides inherent robustness. Erroneous depth measurements (outliers) are often inconsistent across frames and thus accumulate lower confidence.

• Weighting Function: Typically decreases with distance from the sensor's optical center or with high depth measurement uncertainty.
• Hole Filling: The fusion process can fill holes present in individual depth maps if other views provide data for that region.

A Signed Distance Function (SDF) is a continuous scalar field that defines, for any point in 3D space, the shortest distance to the surface of an object, with the sign indicating whether the point is inside (negative) or outside (positive) the object. It is the foundational mathematical concept from which the TSDF is derived.

• Core Property: The zero-level set of an SDF (where the distance value is zero) defines the object's surface.
• Continuity: Provides smooth, differentiable information about the scene geometry, which is advantageous for optimization.
• TSDF Relationship: A TSDF is a practical, discretized implementation of an SDF where distance values are truncated (clamped) to a fixed range around the surface for computational efficiency and robustness to noise.

A voxel grid is a regular, volumetric discretization of 3D space into a structured lattice of cubic elements called voxels. It is the most common data structure for storing a TSDF.

• Structure: Each voxel stores one or more values. In a TSDF grid, this is typically the truncated signed distance and a weight for running averages.
• Memory vs. Resolution: The memory requirement scales with O(n³), making high-resolution grids computationally expensive. This led to the development of sparse structures like Octrees.
• Fusion Process: Depth maps from multiple viewpoints are integrated by updating the TSDF values in the corresponding voxels, progressively building a consensus model of the surface.

KinectFusion is a seminal real-time 3D reconstruction system introduced by Microsoft Research that popularized the use of TSDFs for dense volumetric mapping. It used a commodity RGB-D sensor (Microsoft Kinect) to create dense 3D models of indoor environments.

• Core Innovation: Demonstrated real-time, incremental TSDF fusion and surface prediction using GPU acceleration.
• Pipeline: The system continuously 1) captures depth frames, 2) estimates camera pose (via ICP on the predicted surface), 3) integrates the new depth data into a global TSDF volume, and 4) uses Ray Casting to render the current model for tracking and visualization.
• Impact: It set the standard for real-time dense SLAM and directly influenced subsequent systems like DynamicFusion and ElasticFusion.

Marching Cubes is a classic computer graphics algorithm for extracting a polygonal mesh surface from a 3D scalar field, such as a TSDF volume. It is the standard method for converting a reconstructed TSDF into a renderable mesh.

• Process: The algorithm 'marches' through each cell (8 adjacent voxels) in the volume. Based on the sign (inside/outside) of the scalar values at the 8 corners, it looks up a pre-defined table to generate the set of triangles that approximate the surface within that cell.
• Iso-surface: It extracts the iso-surface where the scalar value equals zero (the zero-crossing of the SDF).
• Efficiency: It generates a watertight mesh but can produce a high polygon count, often requiring subsequent mesh simplification.

In the context of TSDF volumes, Ray Casting is the primary technique for rendering the implicit surface directly from the volumetric representation without first extracting a mesh. It is crucial for real-time visualization and camera tracking.

• Mechanism: For each pixel, a ray is shot from the camera into the volume. The algorithm samples TSDF values along the ray, looking for the zero-crossing (where the sign changes from positive to negative).
• Surface Prediction: The precise 3D location of the zero-crossing is calculated, often via linear interpolation, providing a depth value and a surface normal for that pixel.
• Use in SLAM: This predicted depth map and normal map are directly compared to the live sensor input for iterative closest point (ICP) camera tracking, as used in KinectFusion.

RGB-D Reconstruction is the overarching process of creating a 3D model from a sequence of RGB-D (color plus depth) images. TSDF-based fusion is the dominant volumetric method for this task.

• Sensor Input: Relies on active depth sensors like Microsoft Kinect, Intel RealSense, or Structured Light scanners that provide synchronized color and depth per pixel.
• Pipeline Stages: A typical pipeline involves 1) sensor data capture, 2) camera pose estimation (via Visual SLAM or ICP), 3) volumetric fusion (using a TSDF), and 4) surface extraction (using Marching Cubes).
• Contrast with MVS: Unlike Multi-View Stereo (MVS), which computes depth from multiple color images, RGB-D reconstruction starts with measured depth, making it more robust in textureless regions but limited by sensor range and noise characteristics.