Dynamic NeRF (Neural Radiance Field)

This dominant architecture learns a static canonical radiance field representing the scene in a reference pose. A separate time-conditioned deformation field maps observed 3D points at time t back to their canonical coordinates. This separation simplifies learning:

• Key Insight: Appearance is modeled in a stable canonical space.
• Deformation Field: A small MLP predicts a 3D displacement vector (Δx, Δy, Δz) for each (x, y, z, t) query.
• Examples: D-NeRF, Nerfies. Ideal for smoothly deforming objects like cloth or faces.

A direct extension where the NeRF MLP takes time as an additional input coordinate. The network F_Θ(x, y, z, θ, φ, t) → (c, σ) must learn to disentangle spatial and temporal variations implicitly.

• Architecture: The same core MLP processes all inputs.
• Challenge: Prone to overfitting to training views and times without strong regularization.
• Use Case: Best for scenes with global, coherent temporal changes like lighting transitions.
• Regularization: Often requires a temporal smoothness loss to prevent flickering.

This architecture explicitly models 3D motion by jointly learning a static NeRF and a scene flow field. For a point at time t, it predicts where it will move to by time t+1.

• Dual Output: F_Θ(x, y, z, t) → (c, σ, flow) where flow is a 3D motion vector.
• Cycle Consistency: A core loss ensures flow predictions are temporally consistent forward and backward.
• Benefit: Provides interpretable 3D motion vectors, useful for robotics and scene understanding beyond novel view synthesis.

An explicit, rasterization-based alternative to implicit NeRFs. Represents a dynamic scene with a set of 3D Gaussians whose attributes are functions of time.

• Attributes: Position, rotation, scale, opacity, and spherical harmonics coefficients are all f(t).
• Performance: Enables real-time rendering of dynamic scenes, a significant advantage over most implicit methods.
• Training: Uses a differentiable splatting renderer and standard gradient descent.
• Trade-off: Higher memory footprint for storage but extremely fast inference.

Incorporates recurrent network units (e.g., LSTMs, GRUs) into the NeRF architecture to model temporal dependencies across a sequence.

• Mechanism: The hidden state of the RNN cell acts as a memory of previous scene states.
• Advantage: Can model longer-term dynamics and dependencies better than frame-by-frame methods.
• Application: Well-suited for sequential video input where the order matters, such as forecasting future frames.
• Complexity: More challenging to train due to the recurrent architecture's stability requirements.

Models dynamic geometry using an implicit surface representation that varies over time. The network learns a function SDF(x, y, z, t) → s, where s is the signed distance to the surface at that moment.

• Rendering: Typically uses a differentiable sphere tracing renderer instead of volume rendering.
• Benefit: Provides high-quality, watertight surface geometry at each timestep.
• Extension: Often combined with a separate time-varying appearance network for texture.
• Example: Used in dynamic reconstruction of rigidly moving objects with clear surfaces.

4D reconstruction is the foundational computer vision task of creating a time-varying, dynamic 3D model from a sequence of images or videos. It extends traditional 3D reconstruction by capturing the temporal evolution of geometry and appearance. This is the overarching goal that methods like Dynamic NeRF aim to solve.

• Core Output: A spatio-temporal model (3D + time).
• Primary Data Source: Multi-view video or monocular video sequences.
• Key Challenge: Maintaining temporal coherence while handling occlusions and novel motion.

Deformable NeRF is a specific architectural approach within Dynamic NeRF. It models non-rigid motion by learning a continuous deformation field. This field maps points from a learned canonical space (a static 3D representation) to their observed positions at each timestep.

• Core Mechanism: A neural network that outputs a 3D displacement vector for any (x, y, z, t) coordinate.
• Advantage: Separates learning of canonical appearance/geometry from complex motion.
• Use Case: Ideal for modeling elastic objects like clothing, fluids, or soft bodies.

Neural Scene Flow Fields (NSFF) is a seminal method that jointly learns a dynamic radiance field and a 3D scene flow field from monocular video. Scene flow is the 3D motion vector of every point in the scene. NSFF explicitly regularizes the learned motion to be cycle-consistent (moving a point forward and then backward in time should return it to its start).

• Key Innovation: Unsupervised learning of dense 3D scene flow from video alone.
• Output: Enables both novel view synthesis and motion estimation.
• Limitation: Assumes mostly static cameras or known camera motion.

4D Gaussian Splatting is an explicit, point-based alternative to implicit Dynamic NeRF models. It represents a dynamic scene with a set of 3D Gaussians whose attributes—position, rotation, scale, opacity, and spherical harmonics coefficients—are defined as continuous functions of time. This leverages the rasterization-based efficiency of 3D Gaussian Splatting for real-time performance.

• Representation: Explicit set of animated 3D Gaussians.
• Primary Benefit: Enables real-time rendering of dynamic scenes.
• Trade-off: Less compact than implicit representations; requires storing parameters for many Gaussians over time.

A temporal coherence loss is a critical regularization term used when training dynamic scene representations. It penalizes unrealistic or abrupt changes in geometry, appearance, or motion between consecutive timesteps. This loss is essential for producing smooth, physically plausible reconstructions from sparse or noisy video data.

• Function: Enforces smoothness over time in the learned model.
• Common Forms: Penalizes large accelerations in scene flow, or large differences in color or density for the same world point across nearby frames.
• Result: Reduces flickering and instability in synthesized novel views.

Canonical space mapping is a core strategy in deformable reconstruction. All observations of a deforming object are mapped back to a single, fixed reference configuration (the canonical space). The neural network then only needs to learn one set of appearance and density parameters in this stable space, while a separate network learns the time-dependent deformation to and from it.

• Analogy: Similar to "unposing" a character in animation to edit its base model.
• Benefit: Simplifies learning, as appearance is disentangled from complex motion.
• Challenge: Requires learning an invertible deformation field to map between canonical and observed spaces.