Dynamic NeRF

The most fundamental approach treats time as an additional input coordinate to the neural network, alongside 3D spatial location (x, y, z) and viewing direction. The multilayer perceptron (MLP) learns a 5D function: f(x, y, z, θ, φ, t) → (c, σ). This allows the model to represent a 4D spatiotemporal volume, where density and color can change continuously over time. However, this simple formulation can struggle with complex motions and often requires extensive, dense temporal sampling.

A more structured approach introduces a deformation field that maps points from an observed spacetime (x, t) back to a canonical, or rest, space. The architecture typically consists of two networks:

• Deformation Network: Predicts a displacement vector: T(x, t) → Δx.
• Canonical NeRF: A standard NeRF that models the static scene in canonical coordinates: f(x_c, d) → (c, σ). Rendering involves transforming each sampled 3D point along a ray at time t back to the canonical frame before querying the canonical NeRF. This disentangles appearance from motion, improving generalization for cyclic or rigid motions.

This method explicitly models the 3D motion vector (scene flow) for every point in space and time. The network outputs not only color and density but also a flow vector v that describes how that point moves to the next time step. This is crucial for tasks beyond view synthesis, such as:

• Frame interpolation: Generating novel views at unseen timestamps.
• Motion segmentation: Differentiating between independently moving objects.
• Future prediction: Extrapolating scene dynamics. Training often requires additional constraints like flow consistency losses to ensure physically plausible motion.

This conceptual framework models the full plenoptic function over time. Architectures like DyNeRF or NeRFPlayer treat a dynamic scene as a continuous function from (x, y, z, θ, φ, t, λ) to radiance. Key engineering challenges include:

• Memory efficiency: Storing a 4D field is prohibitive. Solutions use tensor factorization (e.g., decomposing space and time into compact low-rank tensors) or time-aware hash grids (extending Instant NGP's multi-resolution hash encoding to 4D).
• Temporal coherence: Avoiding flickering by ensuring smooth transitions between frames, often enforced via temporal smoothness regularization in the loss function.

Instead of feeding time t directly, a latent code z_t can be learned to represent the state of the scene at each frame or time interval. This latent vector is concatenated with the spatial inputs to the NeRF MLP. Benefits include:

• Disentanglement: The latent space can capture complex, non-rigid motions more compactly than a continuous time variable.
• Compression: The sequence of latent codes provides a compressed representation of the dynamic scene.
• Control: Interpolating or manipulating latent codes allows for temporal editing and motion synthesis. This approach is common in models trained on datasets of similar object categories (e.g., talking faces).

For scenes with multiple independent moving objects, a monolithic Dynamic NeRF is insufficient. Neural scene graphs or object-centric architectures are used, where:

• Each object is modeled by its own local Dynamic NeRF.
• A compositional rendering process composites them using learned or estimated transformation matrices (rotation, translation) over time.
• This requires solving the challenging problems of object discovery, tracking, and decomposition from 2D videos, but enables powerful editing capabilities like independent object manipulation, removal, or re-timing.

The foundational technique upon which Dynamic NeRF is built. A standard NeRF represents a static 3D scene as a continuous volumetric function, using a multilayer perceptron (MLP) to map a 3D coordinate and viewing direction to a volume density and view-dependent color. It is optimized via differentiable volume rendering to synthesize photorealistic novel views from a set of posed 2D images.

The core computer vision task that NeRF and Dynamic NeRF address. It involves generating a photorealistic image of a scene from an arbitrary camera viewpoint that was not present in the original input set. Dynamic NeRF specifically tackles the challenge of temporal view synthesis, generating novel views at arbitrary moments in time for scenes with motion.

A structured, hierarchical representation for complex or dynamic scenes. Instead of a single monolithic NeRF, a scene is decomposed into objects, each represented by its own local neural field (e.g., a small NeRF or SDF). These objects are connected via spatial transformations (translation, rotation) within a graph. This is highly relevant for Dynamic NeRF as it provides a natural framework for modeling independent object motion and enabling compositional editing.

An alternative, non-neural approach to creating dynamic 3D models. It uses arrays of synchronized cameras to record a subject (often a person) from all angles, producing a time-varying 3D volume (like a 3D video). While Dynamic NeRF infers a continuous scene representation from sparse views, volumetric capture directly measures it from dense camera arrays, making it data-rich but hardware-intensive. The outputs are often used as training data for dynamic neural representations.

The end-user application enabled by technologies like Dynamic NeRF and volumetric capture. It refers to interactive video where the viewer can dynamically choose the camera angle during playback, as if controlling a virtual camera moving around the action. Dynamic NeRF is a leading method for generating free-viewpoint video from conventional, sparse camera rigs by learning a continuous spatio-temporal scene model.

The standard optimization paradigm for most NeRF models, including many Dynamic NeRFs. Also called per-scene optimization, it involves training a model (often from scratch) on the specific set of images and camera poses for a single scene. This contrasts with a generalizable NeRF that works across scenes instantly. Dynamic NeRFs frequently use this approach, where the network parameters defining the scene's geometry, appearance, and motion are all optimized for that one dynamic sequence.