4D Gaussian Splatting

Unlike implicit neural representations like Dynamic NeRF, 4D Gaussian Splatting uses an explicit, point-based model. Each scene element is represented by a collection of 3D anisotropic Gaussians—ellipsoids defined by a center (mean), a 3x3 covariance matrix (defining scale/rotation), opacity, and spherical harmonic coefficients for view-dependent color. The critical 4D extension is that all these attributes (position, rotation, scale, opacity, color) are parameterized as continuous functions of time, typically using compact neural networks or polynomial bases. This allows for smooth interpolation and rendering at any arbitrary timestamp within the captured sequence.

The core rendering operation is differentiable splatting. To render an image from a given camera pose and time:

• Each 3D Gaussian is projected onto the 2D image plane, becoming a 2D Gaussian.
• Pixels are rasterized by alpha-blending these 2D splats in depth order.
• Crucially, this entire process is differentiable, enabling gradient-based optimization. The renderer can backpropagate gradients from a photometric loss (comparing rendered pixels to captured video frames) to update all Gaussian parameters (positions, covariances, colors, opacities) and their temporal evolution functions. This allows the model to be trained from multi-view video without explicit 3D supervision.

A primary advantage over neural volumetric methods is real-time rendering performance, even for dynamic scenes. The explicit representation enables highly optimized rasterization pipelines:

• Fast Culling & Sorting: Gaussians outside the view frustum or fully opaque can be efficiently culled. Depth sorting is accelerated using tile-based rasterization.
• GPU Parallelization: The independent nature of splats allows for massive parallelization on modern graphics hardware.
• This enables interactive frame rates (60+ FPS) for complex dynamic scenes, which is critical for applications in augmented reality, virtual reality, and dynamic free-viewpoint video.

To model motion, 4D Gaussian Splatting often employs a canonical space strategy combined with a deformation field. A set of canonical, static 3D Gaussians represents the scene's base geometry and appearance. A separate neural network (the deformation field) takes a canonical 3D point and a time value as input and outputs a displacement vector and other attribute deltas (e.g., rotation change). This maps the canonical Gaussians to their observed positions and states at each moment. This approach enforces temporal coherence and is more efficient than modeling each frame independently, as it shares learned appearance details across time.

The model dynamically manages its complexity through adaptive density control during training. This process involves:

• Cloning: Gaussians in areas with large positional gradient magnitude (under-reconstruction) are cloned to increase detail.
• Pruning: Gaussians with very low opacity are periodically removed.
• This allows the representation to start sparse and automatically increase its density in regions of complex geometry or high-frequency appearance, leading to a memory-efficient and detail-adaptive reconstruction. For dynamic scenes, this control must operate consistently over time to avoid flickering artifacts.

The representation offers a favorable trade-off between quality and storage. A dynamic scene is encoded not as a sequence of dense meshes or voxel grids, but as:

• A set of canonical Gaussian parameters (position, covariance, SH coefficients).
• The relatively small weights of the deformation field MLP that governs motion.
• This results in a compact file size compared to uncompressed 4D mesh sequences. The representation is also highly compressible, as the Gaussian parameters are well-suited for traditional quantization and entropy coding techniques, facilitating streaming or storage of 4D content.

Creates high-fidelity, dynamic digital replicas of real-world environments (e.g., factories, cities) that evolve in sync with their physical counterparts. Key uses include:

• Predictive maintenance: Simulate equipment wear and failure scenarios.
• Urban planning: Model pedestrian and vehicle traffic flow.
• Training & safety: Run simulations for autonomous vehicles or robots in a photorealistic, dynamic virtual world before real-world deployment.

Drives next-generation AR/VR experiences by enabling real-time rendering of complex, dynamic 3D content anchored to the physical world.

• Social AR: Render expressive, dynamic avatars of remote participants into a user's physical space.
• Industrial AR: Overlay animated assembly instructions or real-time machine telemetry onto physical equipment.
• Immersive storytelling: Allow users to walk around and interact with dynamic narrative scenes.

Provides robots with a dense, temporally coherent 3D understanding of dynamic environments for navigation and manipulation.

• Scene dynamics prediction: Forecast future states of moving objects (e.g., pedestrians, other vehicles).
• Deformable object manipulation: Model the 3D shape and motion of non-rigid objects like cables or fabrics for precise robotic handling.
• Long-term mapping: Maintain an updated 4D map of a changing environment for persistent autonomy.

Revolutionizes content creation pipelines by capturing real-world motion and appearance with unprecedented speed and quality.

• Virtual production: Integrate live-action performances with CG environments in real-time on a LED volume stage.
• High-fidelity asset creation: Rapidly generate animated 3D models of actors, props, or animals from multi-view video.
• Post-production & editing: Modify lighting, materials, or even camera motion in a scene after it has been captured.

Enables the detailed 4D capture and analysis of complex, real-world phenomena that are difficult to instrument directly.

• Biomechanics: Capture and analyze the full-body 3D motion of athletes or patients for performance optimization and rehabilitation.
• Fluid dynamics: Visualize and measure the 3D flow of water, smoke, or air over time from camera arrays.
• Material science: Observe and quantify the deformation and failure modes of materials under stress.

An extension of the foundational Neural Radiance Field (NeRF) framework that models time-varying scenes. Unlike static NeRFs, it incorporates temporal parameters into the neural network, allowing it to represent non-rigid motion and changing appearance. Core approaches include:

• Deformable NeRF: Learns a continuous deformation field mapping points from a canonical 3D space to observed positions per frame.
• Temporal Latent Codes: Uses compact vector embeddings to encode the scene state at each timestep.
• Recurrent Architectures: Employs RNNs or Transformers to model temporal dependencies across sequences.

The computer vision task of calculating a dense 3D motion vector field for every point in a scene. It describes how the observed geometry moves between consecutive frames or timesteps. This is a fundamental problem underpinning dynamic reconstruction, as it provides the motion compensation needed to align observations. Techniques range from traditional optimization to deep learning models that predict flow directly from point clouds or images.

The overarching process of creating a time-varying 3D model from a sequence of images or video. The output is a 4D representation (3D space + time) that captures both geometry and its evolution. This is the primary goal that methods like 4D Gaussian Splatting address. Key challenges include maintaining temporal coherence, handling occlusions, and decomposing complex motions into rigid and non-rigid components.

The task of generating photorealistic images of a dynamic scene from arbitrary, unseen viewpoints and timestamps. This is the primary rendering objective for 4D representations. It requires the model to accurately interpolate and extrapolate both appearance and motion. Success is measured by the visual quality and temporal smoothness of the synthesized novel-view video sequences.

A fundamental dichotomy in 3D/4D modeling. 4D Gaussian Splatting is an explicit representation, using a discrete set of 3D Gaussians with defined attributes. In contrast, implicit representations like NeRFs or Dynamic SDFs use a neural network as a continuous function to query properties at any coordinate. Key trade-offs:

• Explicit: Faster rendering, easier to edit, but memory scales with scene complexity.
• Implicit: Compact, memory-efficient for empty space, but rendering requires many network queries.

Specialized domains within dynamic reconstruction that impose strong motion priors. Articulated motion models represent objects (e.g., humans, robots) as a kinematic chain of rigid parts, using skinning weight networks to deform a surface. Human/Facial Performance Capture aims for high-fidelity 4D reconstruction of people, often requiring specialized multi-view systems and models that capture subtle expressions and wrinkles.