Dynamic NeRF

A 4D neural field is the core architectural concept for Dynamic NeRF, where the neural network maps a 3D spatial coordinate (x, y, z) and a time value (t) to volumetric density and view-dependent color. This creates a continuous spatio-temporal representation, allowing the model to query the scene state at any moment. The network must learn to disentangle static geometry from dynamic motion, a significant challenge that often requires explicit architectural inductive biases or regularization.

A common approach to modeling motion without a monolithic 4D field is to use a canonical space and a deformation field. The scene's geometry and appearance are defined in a static canonical 3D NeRF. A separate neural network (the deformation field) learns to map a 3D point at time t back to its corresponding position in the canonical space. This is computationally efficient for repetitive or constrained motions, as it reuses the canonical representation.

• Example: D-NeRF uses an MLP to predict a 3D displacement vector for every (x, y, z, t) query.

Inspired by NeRF-W's handling of varying illumination, some dynamic models use per-frame or per-time latent codes to capture transient changes. Instead of modeling geometric motion, these appearance embeddings allow the static geometry network to modulate its color output based on a time-index. This is effective for scenes with dynamic lighting, weather effects, or the presence of transient objects that do not alter the underlying 3D structure.

Neural scene flow refers to the dense, per-point 3D motion vector field predicted by a Dynamic NeRF model. It represents the velocity of every scene point over time. Architectures that explicitly predict flow, such as NSFF, regularize this flow to be piecewise smooth and often enforce cycle consistency (warping a point forward and then backward in time should return it to the start). This provides an interpretable intermediate representation of dynamics and improves reconstruction quality.

Rendering a dynamic scene requires integrating along rays in both space and time. Space-time ray sampling strategies are critical. Instead of a static camera ray, the ray origin/direction may be a function of time (e.g., for a moving camera). The volume rendering integral is extended to accumulate color and density from samples taken along this spatio-temporal path. Efficient, adaptive sampling in this 4D domain is more complex than in static NeRF and is an active area of research.

To handle the increased complexity of 4D data, many state-of-the-art Dynamic NeRF methods adopt hybrid representations. They combine the implicit continuity of neural fields with explicit, efficient data structures for time. Examples include:

• 4D hash grids (extending InstantNGP to 4D).
• Tensor factorizations across spatial and temporal dimensions.
• Voxel grids with time-varying features. These structures drastically reduce the burden on the MLP, enabling faster training and real-time rendering of dynamic content.

Dynamic NeRF creates temporally-aware digital twins of real-world environments, capturing not just geometry but also how it evolves. This is critical for high-fidelity simulations in robotics and autonomous systems.

• Training Data for Embodied AI: Provides photorealistic, dynamic environments for training vision-language-action models in simulation before sim-to-real transfer.
• Predictive Modeling: By learning the dynamics of a scene (e.g., people moving in a warehouse), it can be used to generate plausible future states for predictive planning.
• Infrastructure Monitoring: Models changes in industrial sites or construction projects over time for progress tracking and anomaly detection.

In spatial computing, Dynamic NeRF enables persistent, dynamic AR overlays and highly realistic VR environments. It bridges captured reality with interactive digital content.

• Persistent AR: A Dynamic NeRF of a room can be anchored to the physical space, allowing virtual objects to interact consistently with real-world objects that may move (e.g., a virtual character walking around a real chair).
• Volumetric Video in VR: Creates immersive experiences where users can move around within a captured dynamic event, such as a sports game or concert.
• Real-Time Requirement: Drives research into real-time neural rendering and acceleration techniques like InstantNGP for 4D fields.

Dynamic NeRF provides a powerful tool for non-invasive, high-resolution 4D capture in scientific imaging and medical diagnostics, where understanding motion is crucial.

• Biomechanics Analysis: Capturing the 3D motion of organisms or human gait for detailed kinematic studies without intrusive markers.
• Medical Imaging Synthesis: Potentially generating dynamic 3D visualizations from limited 2D medical imaging slices (e.g., MRI, ultrasound) over time.
• Microscopy: Reconstructing 3D dynamics of live cell cultures or microscopic processes from a focal stack over time.

By decomposing a dynamic scene into time-varying geometry, appearance, and lighting, Dynamic NeRF enables post-capture manipulation that was previously impossible with 2D video.

• Temporal Object Removal: Removing a moving person or vehicle from a scene across all frames and viewpoints, filling in the static background convincingly.
• Re-timing & Slow Motion: Generating novel temporal frames to create smooth slow-motion effects from multi-view video.
• Dynamic Relighting: Changing the illumination of the captured dynamic scene to match a new virtual environment, requiring separation of intrinsic albedo from transient shadows.

For embodied intelligence systems, Dynamic NeRF provides a rich world model that predicts not just the static layout but the future state of dynamic agents, which is essential for safe navigation and manipulation.

• 4D Occupancy Forecasting: Predicting the future 4D occupancy grid of a scene (e.g., a street intersection) to plan safe robot or vehicle trajectories.
• Improved SLAM: Systems like NeRF-SLAM are extended to dynamic environments, where the model must simultaneously track the camera, map the static world, and identify/segment moving objects.
• Human-Robot Interaction: Creating a real-time dynamic model of a human collaborator to enable more natural and responsive robotic assistance.

A 4D neural field is a continuous function, typically represented by a neural network, that maps a 3D spatial coordinate (x, y, z) and a time value (t) to scene properties like color and density. This is the foundational mathematical representation for Dynamic NeRF, enabling the modeling of deformable geometry, moving objects, and temporal phenomena like fluid flow. Unlike a static 5D NeRF field, the 4D field treats time as an explicit input dimension, allowing for querying the scene state at any moment.

A deformation field is a common architectural component in Dynamic NeRF that models scene motion by warping points from a canonical, static 3D space into an observed, deformed space at a given time. The network typically learns two functions:

• A canonical NeRF that defines geometry and appearance in a reference frame.
• A deformation network that predicts a 3D displacement vector for any (x, y, z, t) coordinate. This separation simplifies learning by factoring time-varying motion from time-invariant appearance, making it effective for modeling articulated or non-rigid deformations.

NeRF in the Wild (NeRF-W) is a pivotal model for handling unstructured, dynamic photo collections. While not strictly a 4D model, it introduces key concepts for separating static and transient elements, which is a precursor to full dynamic modeling. Its core innovations are:

• Appearance Embeddings: Per-image latent vectors that capture variable lighting and atmospheric conditions.
• Transient Uncertainty Estimation: A separate network head that predicts per-point uncertainty, allowing the model to down-weight transient objects (like people, cars) and reconstruct a clean, static background scene. This work demonstrated that NeRFs could be robust to real-world, uncontrolled data with inherent dynamics.

NeRF-SLAM is a class of systems that perform Simultaneous Localization and Mapping (SLAM) by using a NeRF as the map representation. For dynamic scenes, these systems must differentiate between static infrastructure and moving objects to achieve accurate localization. Key technical challenges include:

• Dynamic Object Segmentation: Identifying and masking out non-static elements in real-time to prevent map corruption.
• Joint Pose and Map Optimization: Continuously updating both the camera trajectory and the implicit neural map as new video frames stream in. This application is critical for autonomous navigation in environments with pedestrian and vehicular traffic.

Bundle-Adjusting NeRF (BARF) is a method for training a NeRF with imperfect or unknown camera poses. For dynamic scene capture, accurate per-frame camera estimation is crucial but often unavailable. BARF enables joint optimization of:

• The neural scene representation (geometry and appearance).
• The 6-Degree-of-Freedom (position and orientation) camera pose for every input frame. It achieves this by using a coarse-to-fine scheduling of the positional encoding frequencies, which prevents the optimization from getting stuck in poor local minima. This is essential for creating Dynamic NeRFs from handheld or moving camera videos.

Plenoptic video refers to the capture and synthesis of a time-varying light field—the full 7D plenoptic function (3D position, 2D direction, 1D wavelength, 1D time). Dynamic NeRF is a leading technique for modeling plenoptic video from sparse camera arrays. The goal is to enable 6-Degree-of-Freedom (6DoF) video playback, where a viewer can change their viewpoint and move through time freely. This has direct applications in:

• Volumetric video conferencing and immersive replay for sports.
• Creating dynamic digital twins of industrial sites or natural phenomena.
• Free-viewpoint video for entertainment and virtual production.