Dynamic View Synthesis

The core challenge is representing a scene as a continuous function of 3D space (x, y, z) and time (t). Unlike static NeRFs, dynamic models map these coordinates to radiance (color) and volume density, capturing how geometry and appearance change. Common representations include:

• 4D Neural Radiance Fields: A single MLP takes (x, y, z, t) as input.
• Canonical Space + Deformation Field: A static NeRF exists in a canonical pose, with a separate network predicting a 3D motion vector for each point at time t.
• Explicit 4D Gaussians: Attributes like position, rotation, and color are modeled as continuous functions of time.

To avoid overfitting and enable editing, models must separate appearance from motion. This is achieved through specialized architectures:

• Deformation Fields: Learn a continuous, smooth mapping from observed coordinates at time t back to a canonical, static 3D space. This isolates the object's inherent color and shape.
• Articulated Models: For human/robot capture, skinning weight networks predict how 3D points are influenced by a skeleton's joints.
• Rigid Motion Decomposition: The scene is segmented into components that move as rigid bodies, simplifying motion estimation. Failure to properly disentangle leads to blurring or ghosting artifacts in novel views.

Generated novel views must be physically plausible across time. Models enforce this via:

• Temporal Coherence Losses: Regularization terms that penalize unrealistic, abrupt changes in geometry or color between consecutive frames.
• Scene Flow Estimation: Many methods jointly learn a 3D motion vector field (scene flow) alongside radiance, ensuring motion is consistent across all spatial points.
• Recurrent Architectures (RNR): Use LSTMs or GRUs to maintain a hidden state across timesteps, explicitly modeling dependencies on past observations. This is crucial for handling occlusions and predicting future states.

Performance heavily depends on the training data:

• Multi-View Video: The gold standard. Synchronized videos from many cameras provide strong geometric constraints at each timestep.
• Monocular Video: A far more challenging, under-constrained setting. It requires strong motion priors (e.g., smoothness, rigidity) to resolve ambiguity. Methods like Neural Scene Flow Fields (NSFF) pioneer this domain.
• Sparse View Input: Some methods aim to learn dynamic scenes from very few camera angles per timestep, leveraging information sharing across time to compensate for missing spatial views.

The rendering equation integrates over the 4D representation. For a given virtual camera pose and requested time t', the model:

1. Samples 3D points along camera rays.
2. Queries the dynamic scene representation at those (x, y, z) coordinates and time t' for density and color.
3. Differentiably composites these samples into a 2D pixel color, using the same volume rendering integral as static NeRF. This allows end-to-end training from only 2D image and time stamps, without 3D or flow supervision. The gradient flows back to update both geometry and motion parameters.

A trained dynamic view synthesis model enables several advanced capabilities beyond static novel view synthesis:

• Dynamic Free-Viewpoint Video: Render the scene from any viewpoint and any moment in time, enabling virtual camera navigation in 4D.
• Temporal Super-Resolution & Frame Interpolation: Generate smooth, high-frame-rate video by rendering at intermediate, non-captured timestamps.
• Motion Editing & Re-timing: Alter the speed or path of object motion by manipulating the time input or deformation field.
• 4D Semantic Segmentation: Assign class labels (e.g., 'car', 'pedestrian') to 3D points consistently across time.
• Human Performance Capture: Create high-fidelity 4D avatars from multi-view video for film, VR, and telepresence.

Dynamic NeRF extends the original Neural Radiance Field framework to model scenes with motion. It incorporates time as an additional input coordinate to the neural network, allowing it to output different density and color values for the same spatial point at different moments. This enables the synthesis of novel views at arbitrary, unseen timestamps within a captured sequence. Core approaches include:

• Temporal NeRF: Explicitly models time as a continuous variable.
• Deformable NeRF: Learns a continuous deformation field that maps points from a canonical static space to observed positions at each frame.
• Recurrent NeRF (RNR): Uses recurrent neural network layers to model temporal dependencies across frames.

4D Gaussian Splatting is an explicit, point-based alternative to implicit neural fields for dynamic scenes. It represents a scene as a set of 3D Gaussians whose attributes—position, rotation, scale, opacity, and spherical harmonic coefficients for color—are defined as continuous functions of time. This representation is highly efficient for real-time rendering and naturally models dynamic phenomena like splashing water or fluttering cloth. Training uses a differentiable rasterizer, and the temporal evolution of Gaussian parameters is often controlled by a compact neural deformation field or learned motion bases.

Neural Scene Flow Fields (NSFF) is a method that jointly learns a time-varying neural radiance field and a 3D scene flow field from monocular video. The scene flow field assigns a 3D motion vector to every point in space and time, describing its trajectory. This coupling allows the model to reason about occlusion and disocclusion over time, leading to more physically plausible novel view synthesis for dynamic scenes. It addresses the challenge of temporal consistency without requiring pre-computed camera poses or depth maps, learning everything end-to-end from video frames.

Scene flow estimation is the foundational computer vision task of calculating the 3D motion vector field for every point in a scene between two time steps. It is the 3D extension of 2D optical flow. In dynamic view synthesis, accurate scene flow is crucial for:

• Motion compensation to align observations across views and time.
• Providing supervision for learning dynamic neural representations.
• Decomposing complex motion into rigid and non-rigid components. Methods range from traditional geometric approaches to deep learning models that directly regress flow from point clouds or multi-view images.

Canonical space mapping is a core strategy in deformable reconstruction where all observations of a non-rigidly deforming object (e.g., a talking face, a walking person) are mapped back to a single, fixed canonical configuration. A neural network learns a deformation field that transforms observed 3D points at time t to their corresponding location in this canonical space. The object's appearance and shape are then learned statically in this canonical space, simplifying the problem. At render time, points are sampled in canonical space, their color/density queried, and then transformed forward to the desired time and viewpoint.

Dynamic free-viewpoint video (DFVV) is the end-user application and output format enabled by dynamic view synthesis. It allows a viewer to interactively and seamlessly change both the virtual camera's viewpoint and the playback time within a reconstructed dynamic event. This creates an immersive, navigable 4D recording, as if one had a virtual camera rig capturing the event from every angle simultaneously. It is the ultimate goal of systems combining multi-view video processing, 4D reconstruction, and real-time neural rendering, with applications in sports broadcasting, virtual production, and archival preservation of performances.