Dynamic Free-Viewpoint Video

The core of DFVV is a continuous spatio-temporal representation of the scene. Unlike a standard 3D model, this representation encodes geometry, appearance, and motion as a function of 3D coordinates (x, y, z) and time (t). Common implementations include:

• Dynamic Neural Radiance Fields (NeRF): A neural network that maps a 4D coordinate (x, y, z, t) to a volume density and view-dependent color.
• 4D Gaussian Splatting: An explicit representation using anisotropic 3D Gaussians whose attributes (position, rotation, scale, opacity, spherical harmonics) are functions of time.
• Deformation Fields: A vector field that maps points from a canonical 3D space at time t=0 to their observed positions at any time t, simplifying the learning of a static appearance model.

High-quality DFVV requires synchronized input from multiple calibrated cameras surrounding the subject. This setup provides the necessary visual hull and parallax information to reconstruct 3D geometry at each frame.

• Camera Rig: Typically involves dozens to hundreds of cameras (e.g., the Light Stage, 100+ cameras).
• Synchronization: Hardware genlock ensures all cameras capture within sub-millisecond accuracy.
• Calibration: Intrinsic (focal length, distortion) and extrinsic (position, rotation) parameters for each camera are precisely computed. The plenoptic function is sampled by this camera array.

A DFVV system must model how the scene changes over time in a physically plausible way. This involves estimating scene flow—the 3D motion vector of every point.

• Neural Scene Flow Fields (NSFF): Jointly learns radiance and a continuous 3D flow field from monocular or multi-view video.
• Motion Priors: Constraints like smoothness (neighboring points move similarly) and as-rigid-as-possible deformation are enforced via loss functions.
• Articulated Models: For human subjects, skinning weight networks predict how a skeletal rig deforms the canonical surface, separating pose from identity.

To train the 4D neural representation from 2D images, a differentiable renderer is used. This allows gradients to flow from pixel errors back to the underlying scene parameters (density, color, motion).

• The renderer approximates the volume rendering integral along camera rays, accumulating color and density from the neural field.
• For each training frame, rays are cast from the known camera pose, sampled at the corresponding time t, and rendered into a predicted image.
• The difference between the predicted and captured image (e.g., photometric loss) drives the optimization of the entire 4D model.

The final capability is the interpolation and extrapolation in both viewpoint and time. The trained model acts as a 4D scene database.

• View Synthesis: Given a new camera pose (extrinsic matrix) at a captured time t, the renderer generates the scene from that unseen angle.
• Temporal Interpolation (Frame Interpolation): Querying the model at an intermediate time t' between captured frames generates a novel moment, smoothly interpolating geometry and motion in 3D space, superior to 2D video interpolation.
• Free Navigation: The user can arbitrarily change both the virtual camera's position and the playback time, enabling bullet-time effects and pausing from any angle.

For interactive applications, the massive 4D representation must be optimized for real-time inference.

• Explicit Representations: Methods like 4D Gaussian Splatting are inherently faster for rendering than large MLP queries.
• Model Compression: Techniques like pruning, quantization, and distillation reduce the neural model's size.
• Streaming & Level-of-Detail: For remote viewing, the 4D data is compressed into a video-based format (e.g., multi-view video plus depth, or MV-DDD) or a neural codec, transmitting only the viewpoint- and time-relevant data.

Dynamic NeRF extends the original Neural Radiance Field framework to model scenes with motion. It treats time as an additional input coordinate to the neural network, allowing it to output density and radiance fields that are functions of 3D location, viewing direction, and time.

• Core Mechanism: A multilayer perceptron (MLP) learns a mapping from (x, y, z, θ, φ, t) to (RGB, σ).
• Key Challenge: Requires significantly more data (multi-view video) and compute than static NeRF to disambiguate appearance from motion.
• Example: Modeling a waving flag or a walking person from a ring of cameras.

4D reconstruction is the overarching process of creating a time-varying 3D model (the 4th dimension being time). It is the general goal that dynamic free-viewpoint video achieves as an output format.

• Input Data: Typically multi-view synchronized video streams.
• Output: A coherent 4D sequence of 3D meshes, point clouds, or implicit fields.
• Applications: Sports analytics, scientific visualization of dynamic processes, and historical event preservation.
• Distinction: While dynamic free-viewpoint video emphasizes the interactive viewing experience, 4D reconstruction focuses on the geometric and temporal model itself.

Neural Scene Flow Fields is a seminal method that jointly learns a dynamic radiance field and a 3D scene flow field from monocular video. It explicitly reasons about 3D motion to improve consistency.

• Dual Representation: One network models a static background scene, while another models moving foreground objects and their 3D flow vectors.
• Temporal Ray Sampling: Renders a point at time t by using estimated scene flow to trace its position back to a canonical time t₀ for color/opacity lookup.
• Significance: Demonstrated high-quality dynamic view synthesis from single moving cameras, a highly under-constrained problem.

4D Gaussian Splatting is an explicit, efficient alternative to implicit dynamic NeRFs. It represents a scene as a set of 3D Gaussians whose attributes (position, rotation, scale, opacity, spherical harmonics) are functions of time.

• Explicit Representation: Uses thousands to millions of anisotropic 3D Gaussians as primitives.
• Real-Time Rendering: Leverages differentiable tile-based rasterization, enabling interactive frame rates for dynamic scenes.
• Temporal Modeling: Each Gaussian's attributes are controlled by a compact MLP or a set of keyframe values with interpolation, defining its trajectory and evolution.

Deformable NeRF is a dominant paradigm for dynamic scene modeling that uses a two-stage deformation process. It learns a canonical, template NeRF and a time-dependent deformation field that warps observed points back into this canonical space.

• Canonical Space: A single, static neural radiance field representing the scene's "rest pose" or average state.
• Deformation Field: A second MLP that maps a 3D point at observation time t to a corresponding point in canonical space, plus a residual displacement.
• Advantage: Separates appearance learning (in canonical space) from motion learning, improving generalization and reducing complexity.

Scene flow estimation is the fundamental computer vision task of calculating the 3D motion vector for every point in a scene between two frames. It is a critical component or supervisory signal for many dynamic reconstruction pipelines.

• 3D vs. 2D: While optical flow is 2D pixel motion, scene flow is full 3D motion in world coordinates.
• Role in DFV: Provides dense correspondence across time, enabling temporal coherence, motion compensation, and the alignment of dynamic elements.
• Methods: Ranges from traditional geometric approaches to deep learning models trained on synthetic or real data with LiDAR ground truth.