Recurrent Neural Radiance Fields (RNR)

The core innovation of RNR is the integration of recurrent neural network (RNN) layers—such as Long Short-Term Memory (LSTM) or Gated Recurrent Units (GRU)—into the NeRF architecture. These layers maintain a hidden state vector that is updated sequentially across timesteps. This state acts as a memory, allowing the model to condition its predictions for the current frame on the encoded history of previous observations. This is critical for modeling temporal coherence, ensuring smooth motion and consistent appearance over time, and for handling occlusions, where an object's past states inform its likely appearance when it becomes visible again.

Many RNR implementations adopt a canonical-to-observed space paradigm to disentangle static appearance from dynamic motion. The model learns a single, static canonical neural radiance field that represents the scene in a reference pose. A separate deformation field network—often conditioned on time or a temporal latent code—predicts a 3D displacement vector for each spatial point, mapping it from the canonical space to its observed position at a specific timestep. This separation simplifies learning:

• The canonical field captures permanent texture and geometry.
• The deformation field captures non-rigid motion. This approach is more parameter-efficient than learning a unique radiance field for every frame.

Unlike batch-processing methods, RNR architectures are designed for sequential input processing. The model ingests frames one at a time (or in short sequences), updating its internal state incrementally. This offers several advantages:

• Online Capability: Potential for real-time or streaming reconstruction as new video data arrives, without requiring the entire sequence upfront.
• Long Sequence Modeling: The recurrent mechanism can, in principle, handle arbitrarily long sequences by maintaining a compressed history in its state, though practical limits exist due to memory constraints and vanishing gradients.
• Temporal Super-Resolution: The continuous temporal representation allows for frame interpolation, generating novel views at timestamps between observed frames by querying the model at intermediate time values.

Advanced RNR variants, such as Neural Scene Flow Fields (NSFF), perform joint optimization. They simultaneously learn:

• A dynamic neural radiance field for view synthesis.
• A 3D scene flow field that represents the motion of every 3D point between consecutive frames. This is achieved by incorporating a scene flow loss that enforces consistency between the radiance field rendered at time t and the warped radiance field from time t+1 using the estimated flow. This multi-task learning provides a strong, self-supervised signal that regularizes both geometry and motion, leading to more physically plausible reconstructions from monocular video where 3D motion is ambiguous.

To manage complexity and provide explicit temporal conditioning, RNRs often use temporal latent codes. Each timestep (or short window) is associated with a learnable embedding vector. This code is concatenated with the spatial coordinates when querying the neural network, explicitly telling the model "which moment in time" to represent. For more complex, long-range dependencies, spatio-temporal attention mechanisms can be incorporated. These allow the model to attend to relevant features from specific past frames and spatial regions when reconstructing the current frame, which is particularly useful for modeling periodic motions or interactions that span many frames.

RNR architectures are uniquely suited for scenarios requiring temporal understanding and stateful processing of sequential visual data.

• Monocular Dynamic Reconstruction: Creating 4D models from single moving camera videos, a significantly harder problem than multi-view static reconstruction.
• Free-Viewpoint Video for Dynamic Events: Enabling viewers to navigate spatially and temporally within recorded events like sports or performances.
• Robotics & Autonomous Navigation: Providing a temporally coherent world model for agents, helping predict object trajectories and understand scene dynamics.
• Human & Facial Performance Capture: Modeling the subtle, non-rigid deformations of bodies and faces over time with high fidelity. The key distinction from other Dynamic NeRFs is the explicit architectural bias for sequential data, making RNRs a natural fit for video streams.

Dynamic NeRF is the overarching framework for extending Neural Radiance Fields to model scenes with motion and time-varying appearance. Unlike static NeRF, it incorporates time as an input variable to the neural network. Key approaches include:

• Deformation-based methods that learn a warp from a canonical space.
• Time-conditioned models that directly output color/density for a given (x, y, z, t) coordinate.
• Explicit representations like 4D Gaussian Splatting. RNR is a specific subtype that uses recurrent networks to model temporal state.

Temporal coherence is the principle that scene properties (geometry, appearance, motion) change smoothly and predictably over time. In dynamic reconstruction, enforcing this is critical for realistic results. Techniques include:

• Temporal coherence loss functions that penalize abrupt changes between frames.
• Recurrent networks (like in RNR) that explicitly maintain a hidden state to propagate information.
• Motion priors that assume smooth or periodic motion. Without temporal coherence, models produce flickering, jittering, or physically implausible outputs.

A deformation field is a continuous, learned mapping that defines how points in a static, canonical 3D space move to their observed positions at each timestep. It is central to deformable NeRF approaches. Characteristics:

• It is typically represented by a neural network that outputs a 3D displacement vector for an input (x, y, z, t).
• Allows the model to learn a consistent underlying shape and texture in canonical space, simplifying the learning of appearance.
• Essential for modeling non-rigid motions like cloth deformation or facial expressions.

4D Gaussian Splatting is an explicit, point-based alternative to implicit NeRF representations for dynamic scenes. It models the scene with a set of 3D Gaussians whose attributes (position, rotation, scale, opacity, spherical harmonics coefficients) are defined as continuous functions of time.

• Enables extremely fast, real-time rendering of dynamic scenes.
• Uses differentiable splatting for rasterization.
• The '4D' refers to the 3D spatial dimensions plus time. It represents a shift from dense neural networks to structured, explicit primitives for dynamic content.

Neural Scene Flow Fields (NSFF) is a seminal method that jointly learns a dynamic radiance field and a 3D scene flow field from monocular video. It explicitly reasons about motion:

• Predicts a 3D scene flow vector (motion) for every point in space and time.
• Uses a cycle consistency loss to ensure flow predictions are physically plausible over time.
• Unlike RNR, which uses recurrence, NSFF uses a purely feedforward, time-conditioned network to model dynamics. It bridges view synthesis and 3D motion estimation.

Canonical space mapping is a strategy where all observations of a deforming object are mapped back to a single, fixed reference configuration. This simplifies learning:

• A shared, high-fidelity appearance and geometry model is learned in this canonical space.
• A separate deformation network learns the time-varying mapping to/from this space.
• Benefits include reduced complexity and improved generalization for unseen poses. This approach is foundational to many deformable NeRF and RNR models, providing a stable coordinate system for the core scene representation.