Dynamic 3D Reconstruction

This method represents the evolving geometry of a dynamic scene using a time-varying Signed Distance Function. The surface is defined implicitly as the zero-level set of SDF(x, y, z, t) = 0.

• Volumetric Representation: Captures both the surface geometry and its motion over time within a single continuous function.
• Advantages Over Point Clouds: Provides a watertight, differentiable surface ideal for physics simulation and robotic interaction.
• Optimization: Often trained using depth observations or multi-view RGB video with a differentiable renderer, enforcing consistency across time and viewpoints.

This class of techniques assumes a known or learned template mesh that deforms over time to match observed sensor data. It is prevalent in human body and facial performance capture.

• Process: A pre-defined 3D mesh (the template) is iteratively deformed by optimizing vertex positions to align with input RGB-D frames or multi-view video.
• Regularization: Heavily relies on as-rigid-as-possible (ARAP) or skinning weight priors to ensure deformations are physically plausible and smooth.
• Use Case: The standard for high-fidelity digital human animation and real-time avatar creation in VR/AR.

Scene flow is the 3D motion vector field of every point in a scene. Dense estimation assigns a 3D displacement vector to every voxel or surface point between consecutive frames.

• Relation to Optical Flow: While optical flow is 2D pixel motion, scene flow is its 3D counterpart, describing real-world motion.
• Input Data: Typically computed from stereo or RGB-D video sequences (e.g., from LiDAR or depth cameras).
• Application: Essential for dynamic object tracking, motion segmentation, and providing supervision for self-supervised 4D reconstruction models.

Simultaneous Localization and Mapping (SLAM) systems extended to handle dynamic environments. They explicitly identify and model moving objects while building a map of the static background.

• Core Challenge: Separating static background geometry from dynamic foreground entities in real-time.
• Techniques: Use semantic segmentation (e.g., from a CNN) to mask out potential dynamic objects like people or cars during map building.
• Output: A persistent static map and tracked trajectories for dynamic objects. Critical for autonomous vehicles and robots operating in crowded spaces.

4D reconstruction explicitly refers to the process of generating a time-varying 3D model, where the fourth dimension is time. It is the direct output of dynamic 3D reconstruction pipelines. Key approaches include:

• Temporal neural representations like Dynamic NeRF or 4D Gaussian Splatting that encode scene properties as a function of 3D coordinates and time.
• Non-rigid bundle adjustment that tracks deforming surfaces across a video sequence.
• The result is often a sequence of 3D meshes or a continuous spatio-temporal volume that can be queried at any time instant.

Non-Rigid Structure from Motion is a class of algorithms that extend traditional SfM to recover the 3D shape of deforming objects from 2D video without known point correspondences. It solves the ill-posed problem of factoring 2D trajectories into 3D shape and motion. Core challenges include:

• Shape basis models: Representing deformation as a linear combination of basis shapes.
• Temporal smoothness: Applying priors that deformation changes gradually.
• It is a foundational, often optimization-based, precursor to deep learning methods for dynamic scene capture.

Dynamic NeRF is a neural representation that models a scene's geometry, appearance, and motion simultaneously. A neural network maps a 4D coordinate (x,y,z,t) to color and density. Implementations vary:

• Canonical models: Learn a static 3D representation in a canonical space and a separate deformation field that maps observed points at time t back to that space.
• Time-conditioned models: Directly condition the NeRF MLP on a time embedding.
• These models enable photorealistic novel view synthesis and novel time synthesis from sparse video inputs.

Scene flow is the 3D motion vector field of points in a scene, analogous to optical flow in 2D. Estimating scene flow is critical for segmenting moving objects and understanding dynamics. Methods include:

• RGB-D based: Computing flow from sequences of depth and color images.
• Learning-based: Using convolutional networks to predict flow directly from stereo or monocular video.
• Integration with reconstruction: Scene flow provides dense correspondence across time, which can be used as a constraint in non-rigid bundle adjustment or to regularize dynamic NeRF training.

Volumetric video is the end-product format for dynamic 3D reconstruction, designed for playback in AR/VR and 3D displays. It captures real people or objects in motion as 3D assets. The production pipeline involves:

• Multi-view capture: Using dense arrays of synchronized cameras (e.g., 30+).
• Real-time fusion: Algorithms like Volumetric Performance Capture fuse multi-view RGB-D data into a temporally coherent 4D model.
• Compression & Streaming: Encoding the massive 4D data (textured meshes or neural fields) for efficient transmission, a key challenge for commercial applications.

A Dynamic SDF is an implicit neural representation where a network learns a time-varying signed distance function f(x,y,z,t) = s. The zero-level set of this function defines the deforming surface at each moment. Advantages include:

• Inherent surface topology: Easily handles topological changes (e.g., a hand opening and closing).
• Differentiability: Enables end-to-end training from multi-view video.
• High-fidelity detail: Can capture fine surface details better than explicit mesh sequences. It is a common choice for modeling dynamic, watertight objects like human bodies.