4D Reconstruction

This dominant paradigm extends Neural Radiance Fields (NeRF) by modeling non-rigid motion. A canonical, static 3D scene is defined, and a continuous deformation field maps points from this canonical space to their observed positions at each timestep. This separates the learning of appearance and shape from time-varying motion, leading to higher quality. Key implementations include:

• D-NeRF: Pioneered the use of deformation networks.
• NeRFies: Models subject-specific deformations for faces and bodies.
• HyperNeRF: Uses a higher-dimensional canonical space to handle topological changes.

Moving beyond implicit neural fields, this approach models a dynamic scene as a set of explicit 3D Gaussians whose attributes are functions of time. 4D Gaussian Splatting defines each point's position, rotation, scale, opacity, and spherical harmonic coefficients as continuous over time. Benefits include:

• Real-time rendering speeds, leveraging rasterization pipelines.
• Explicit control over scene elements and motion.
• Efficient modeling of transient phenomena like splashes or smoke. This method bridges traditional computer graphics primitives with differentiable, learnable dynamics.

This technique first estimates 3D scene flow—the motion vector for every point in space between frames—and then integrates it into the reconstruction pipeline. Neural Scene Flow Fields (NSFF) jointly optimize a radiance field and a flow field from monocular video. The process involves:

• Forward/backward flow consistency losses to ensure physically plausible motion.
• Motion compensation to align multi-view observations.
• Rigidity priors to separate static background from moving objects. It is particularly effective for monocular video input where multi-view cues are absent.

For structured objects like humans, animals, or robots, this approach uses a parametric skeletal model. The reconstruction learns:

• Skinning weight networks that predict how mesh vertices are influenced by underlying bones.
• Joint angles and positions over time.
• Per-frame shape and pose parameters (e.g., SMPL model for humans). This provides strong motion priors, ensures anatomical correctness, and enables direct control and animation of the reconstructed sequence. It's the foundation for human and facial performance capture.

This classical, hardware-driven approach uses synchronized camera arrays. It involves:

• Spatio-temporal calibration of dozens of high-speed cameras.
• Per-frame 3D reconstruction using techniques like multi-view stereo or photogrammetry.
• Temporal tracking to establish correspondences between 3D points across frames. While computationally intensive, it produces extremely high-fidelity dynamic free-viewpoint video used in film (e.g., The Matrix) and sports broadcasting. It often serves as ground truth for training data-driven neural methods.

This method compresses the temporal dimension into a compact, learnable representation. The dynamic scene is defined by:

• A shared static geometry/appearance network.
• A set of per-frame or per-time latent codes that modulate the network.
• An auto-decoder framework that optimizes these codes during training. Advantages include:
• Efficient compression of long sequences.
• Easy interpolation and extrapolation in the latent space.
• Separation of motion style from scene content. This approach is common in generative models of dynamic scenes.

An extension of the Neural Radiance Field (NeRF) framework designed to model scenes with non-rigid motion and time-varying appearance. It incorporates temporal parameters into the neural representation, allowing it to output different density and color values for the same spatial coordinate at different times. This is foundational for creating photorealistic, editable 4D content from multi-view video.

• Core Mechanism: The neural network takes inputs of (x, y, z, t, viewing direction).
• Output: Time-conditioned volume density and view-dependent RGB color.
• Challenge: Requires significantly more data and compute than static NeRF to avoid overfitting.

An explicit, point-based representation for dynamic 3D scenes. It models the scene as a set of 3D Gaussian primitives whose attributes—position, rotation, scale, opacity, and spherical harmonic coefficients for color—are defined as continuous functions of time. This method combines the high-quality, real-time rendering of 3D Gaussian Splatting with temporal modeling.

• Advantage: Enables real-time rendering of dynamic scenes at high fidelity.
• Representation: Each Gaussian's trajectory and attribute evolution are often modeled with compact neural networks or polynomial functions.
• Use Case: Ideal for interactive applications like VR and dynamic digital twins.

A method that jointly learns a time-varying neural radiance field and a 3D scene flow field from monocular video. NSFF explicitly models the 3D motion vector of every point in the scene between frames. This dual representation enables both novel view synthesis and motion estimation for dynamic scenes without requiring multi-view camera rigs.

• Key Innovation: Predicts a 3D flow vector (dx, dy, dz) for each spatial point and time.
• Training Constraint: Uses a cycle consistency loss to ensure flow predictions are physically plausible over time.
• Output: Can generate future frames by advecting radiance fields along predicted flow.

A dynamic scene representation that models non-rigid motion by learning a continuous deformation field. This field maps points from an observed, warped space at time t back to a shared canonical, static 3D space. The canonical NeRF then only needs to learn one static appearance and geometry, simplifying the learning problem.

• Two-Stage Process: 1) A network predicts a deformation (x, y, z, t) -> (Δx, Δy, Δz). 2) The canonical NeRF renders the deformed point.
• Benefit: Separates complex motion from static appearance, improving generalization.
• Application: Commonly used for reconstructing talking heads, cloth deformation, and elastic objects.

The computer vision task of calculating the 3D motion vector field for every point in a scene. Unlike 2D optical flow, scene flow describes motion in the world coordinate system (meters/second). It is a fundamental intermediate representation for 4D reconstruction, providing dense correspondence between 3D points across time.

• Input: Typically consecutive frames from a depth sensor (RGB-D) or stereo/multi-view system.
• Output: A vector field where each 3D point in frame t is associated with a 3D displacement to frame t+1.
• Challenge: Requires solving both geometry (3D structure) and correspondence (matching points) simultaneously.

The overarching task of generating photorealistic images of a dynamic scene from arbitrary, unseen viewpoints and timestamps. This is the primary application goal of 4D reconstruction techniques. The quality of synthesis is the key metric for evaluating dynamic NeRF, Gaussian Splatting, and related methods.

• Test Paradigm: Train on multi-view video of a dynamic event, then render novel camera paths not seen during training.
• Requirements: Must maintain temporal coherence (no flickering) and view consistency (geometry looks correct from all angles).
• Industry Use: Creating content for free-viewpoint video, volumetric VR, and visual effects.