Temporal NeRF

This dominant approach learns a continuous, time-varying deformation field that maps observed 3D points at time t back to a shared canonical space. The core NeRF network then only needs to model the static geometry and appearance in this canonical frame. The deformation field, often parameterized by an MLP, accounts for all non-rigid motion, making it highly effective for scenes with complex deformations like cloth or fluids.

• Key Mechanism: T(x, y, z, t) → (Δx, Δy, Δz)
• Benefit: Separates learning of appearance from motion.
• Challenge: Requires careful regularization to prevent the field from becoming degenerate or overly complex.

Instead of deforming space, this architecture directly conditions the radiance field MLP on time. The input is a concatenated vector [x, y, z, θ, φ, t], where t is the temporal coordinate. The network must implicitly learn to associate different spatial regions with different appearances over time.

• Key Mechanism: f_Θ(x, y, z, θ, φ, t) → (c, σ)
• Benefit: Conceptually simple, end-to-end differentiable.
• Challenge: Prone to overfitting on observed timesteps and struggles with temporal interpolation unless the network capacity is very large or specific inductive biases are added.

This approach uses a set of learnable temporal latent codes, one for each training timestep or a learned basis set. The radiance field MLP is modulated by these codes, typically via feature concatenation or AdaIN layers, allowing the scene's state to change without modifying the core network weights.

• Key Mechanism: A latent vector z_t modulates network activations: f_Θ(x, d; z_t).
• Benefit: Efficient; the base network learns shared scene priors while latent codes capture variation.
• Challenge: Requires a method (e.g., an autoencoder) to interpolate or generate latent codes for novel timesteps not seen during training.

Moving beyond implicit networks, this explicit approach models a dynamic scene as a set of 4D Gaussians. Each Gaussian's attributes—3D position, rotation, scale, opacity, and spherical harmonics coefficients for color—are defined as continuous functions of time. Rendering uses tile-based rasterization for real-time performance.

• Key Mechanism: Attributes are functions: μ(t), Σ(t), c(t), α(t).
• Benefit: Enables real-time rendering of dynamic scenes, a significant advance over slow NeRF rendering.
• Challenge: Requires storing and optimizing a large number of primitives; the functional representation of attributes must be carefully designed.

This architecture jointly learns a static NeRF and a 3D scene flow field. The flow field v(x, t) predicts the motion vector for any point in space and time. To render a novel view at time t, points are traced back along the estimated flow to time t=0, where their color and density are queried from the static NeRF.

• Key Mechanism: f_Θ(x, d) → (c, σ) and v_Φ(x, t) → (Δx, Δy, Δz).
• Benefit: Explicitly models 3D motion, which can be a useful output for downstream tasks.
• Challenge: Requires a cycle consistency loss to ensure flow predictions are physically plausible over time.

For sequential video input, architectures incorporate recurrent neural networks (RNNs) like LSTMs or spatio-temporal attention mechanisms. These models maintain a hidden state that aggregates information from previous frames, allowing them to model long-term dependencies and temporal coherence more effectively than frame-by-frame processing.

• Key Mechanism: Hidden state h_t = RNN(h_{t-1}, frame_features_t).
• Benefit: Can handle longer sequences and better model periodic or complex motions.
• Challenge: More difficult to train and render novel views at arbitrary, non-sequential timestamps.

Dynamic NeRF is the overarching category of neural radiance field models that handle non-static scenes. Temporal NeRF is a specific implementation within this category where time is an explicit, continuous input variable. Other approaches include using temporal latent codes or deformation fields to model change.

• Core Distinction: All Temporal NeRFs are Dynamic NeRFs, but not all Dynamic NeRFs use a continuous time parameter.
• Example: A model that uses a separate neural network per frame is a Dynamic NeRF but not a Temporal NeRF.

A Deformable NeRF is a dominant architecture for dynamic scenes that learns a continuous deformation field. This field maps observed 3D points at each timestep back to a shared, static canonical space where appearance and density are defined.

• Mechanism: For a point at time t, the model predicts a displacement vector to warp it into the canonical frame before querying the static NeRF.
• Benefit: Separates learning of persistent scene properties from transient motion, improving generalization.

4D Gaussian Splatting is an explicit, point-based alternative to implicit neural representations like Temporal NeRF. It models a dynamic scene with a set of 3D Gaussians whose attributes (position, rotation, scale, opacity, spherical harmonics) are defined as continuous functions of time.

• Key Advantage: Enables extremely fast, real-time rendering of dynamic scenes, bridging the gap between quality and performance.
• Representation: Each Gaussian's trajectory is typically modeled with low-dimensional temporal latent codes or polynomial functions.

Neural Scene Flow Fields (NSFF) is a seminal method that jointly learns a dynamic NeRF and a 3D scene flow field from monocular video. The scene flow represents the 3D motion vector of every point between frames.

• Joint Optimization: The model is trained with a temporal coherence loss that enforces consistency between the radiance field predicted at time t and the warped field from time t+1 using estimated flow.
• Output: Can synthesize novel views at novel times and also output the estimated 3D motion of the scene.

Dynamic view synthesis is the core computer vision task that Temporal NeRF is designed to solve. It involves generating photorealistic images of a dynamic scene from arbitrary, unseen viewpoints and arbitrary, unseen moments in time.

• Input: Typically multi-view video or monocular video with estimated camera poses.
• Challenge: Requires modeling both the complete 3D geometry/appearance and its continuous temporal evolution, overcoming issues like occlusion and disocclusion over time.

Scene flow estimation is the task of calculating the dense 3D motion vector field for all visible points in a scene between two times. It is a critical supporting task for many dynamic reconstruction methods.

• Relation to Temporal NeRF: Methods like NSFF integrate scene flow estimation directly into the NeRF optimization. Other pipelines use pre-computed scene flow as a motion prior to guide the training of a Temporal NeRF.
• Complexity: More challenging than optical flow, as it requires reasoning about 3D geometry and its change.