3D Gaussian Splatting

The core primitive is a 3D Gaussian ellipsoid, defined by a mean (position), a 3x3 covariance matrix, and an opacity value. The covariance matrix controls the ellipsoid's scale and rotation (anisotropy), allowing it to efficiently represent surfaces, edges, and fine details. This is a key departure from isotropic spherical primitives, enabling a more compact and accurate representation of scene geometry with far fewer elements.

Rendering uses a custom differentiable rasterization pipeline that splats projected 2D Gaussians onto the image plane. The process is:

• Tiling: The screen is divided into tiles (e.g., 16x16 pixels).
• Sorting: Gaussians affecting each tile are sorted by depth.
• Alpha Blending: Gaussians are rendered using fast, approximate alpha-blending along each ray.

This pipeline is orders of magnitude faster than volumetric ray marching used in NeRF, enabling real-time frame rates while remaining fully differentiable for gradient-based optimization.

Each Gaussian stores Spherical Harmonics (SH) coefficients to model view-dependent appearance (e.g., specular highlights). Lower-order SH bands capture diffuse color, while higher-order bands capture finer angular details. This allows the representation to accurately reproduce complex, non-Lambertian materials and lighting effects from a sparse set of input images, a capability inherited from Neural Radiance Fields.

The representation dynamically grows and prunes Gaussians during training to optimize scene coverage. The process involves:

• Cloning Gaussians in areas of high positional gradient (under-reconstruction).
• Splitting large Gaussians to increase detail.
• Pruning Gaussians with very low opacity.

This adaptive mechanism starts from an initial point cloud (from Structure-from-Motion) and automatically densifies it in under-optimized regions, eliminating the need for a predefined, uniform grid or voxel structure.

As an explicit representation, the trained model is simply a list of Gaussian parameters (position, covariance, opacity, SH). This contrasts with the implicit representation of a NeRF, which is a neural network weight set. The explicit structure enables:

• Native GPU friendliness: Rendering maps directly to fast, parallelizable graphics pipelines.
• Real-time performance: Achieves 100+ FPS at 1080p resolution on modern GPUs.
• Easy integration: Can be exported and used in standard graphics engines with a custom shader.

3D Gaussian Splatting addresses key limitations of the original NeRF paradigm:

• Speed: NeRF requires seconds to minutes per frame due to dense ray sampling; Gaussians render in milliseconds.
• Explicitness: NeRF's scene knowledge is locked inside network weights; Gaussians are an editable, structured point cloud.
• Training Time: Gaussian Splatting often trains in minutes, whereas NeRF can take hours.
• Memory: The explicit representation can be more memory-intensive for highly complex scenes but is highly optimized for rendering throughput.

3D Gaussian Splatting is a foundational technology for next-generation AR/VR, enabling photorealistic, real-time rendering of complex environments. Its differentiable rasterization pipeline achieves interactive frame rates (often 60+ FPS) on consumer hardware, a critical requirement for immersive experiences. This allows for:

• Dynamic occlusion where virtual objects correctly interact with reconstructed real-world geometry.
• Six degrees of freedom (6DoF) exploration without pre-baked lighting or geometry limitations.
• Live scene capture and telepresence, where a remote environment can be streamed and explored in real-time.

The technique excels at creating highly accurate digital twins of physical assets—from factory floors to architectural sites. Unlike mesh-based reconstructions, Gaussians capture view-dependent effects and fine details directly from images. Key applications include:

• Facility management and planning: Creating navigable, photorealistic models for maintenance, training, and retrofit planning.
• Cultural heritage preservation: Digitizing fragile artifacts and historical sites with sub-millimeter detail and realistic material appearance.
• Engineering and design review: Allowing stakeholders to visually inspect a photorealistic simulation of a product or environment from any angle before physical construction.

For robotics, 3D Gaussian Splatting provides a dense, explicit 3D scene representation that is more detailed than traditional occupancy grids or point clouds. This supports:

• High-fidelity semantic mapping: Gaussians can be associated with semantic labels (e.g., 'road', 'pedestrian', 'wall') for advanced scene understanding.
• Simulation and testing: Generating realistic sensor data (e.g., camera, LiDAR) from the Gaussian model for training and validating perception algorithms in simulation.
• Path planning in complex environments: The explicit 3D structure allows for precise collision checking and navigation planning around fine geometric details.

In media production, Gaussian Splatting enables novel workflows for volumetric capture and integration. It allows filmmakers to capture real actors or locations as editable 3D assets.

• Virtual production: Placing live actors into digitally rendered environments with correct lighting and perspective interactions.
• Asset creation from video: Turning standard monocular or multi-view video footage into a fully 3D, relightable digital asset for use in CGI scenes.
• Dynamic scene editing: Because the scene is composed of individual, anisotropic Gaussians, artists can selectively edit, remove, or manipulate parts of the reconstructed scene (e.g., deleting an object, changing its color).

The technology revolutionizes online shopping by enabling photorealistic 3D product visualization from a sparse set of images. Consumers can interactively view products from any angle with realistic lighting and materials.

• Virtual try-on and configuration: For furniture, apparel, or consumer electronics, allowing customers to see how a product would look in their space or with different customizations.
• Reduced return rates: By providing a more accurate visual representation than standard 2D images or simple 3D models.
• Scalable asset creation: Automatically generating high-quality 3D views from a product photography turntable setup, bypassing expensive manual 3D modeling.

When applied to aerial or satellite imagery, 3D Gaussian Splatting can generate high-resolution, textured 3D maps of urban and natural landscapes. Compared to traditional photogrammetry (which produces meshes), it offers:

• Superior handling of complex geometry: Such as trees, foliage, and intricate building facades, which are challenging for mesh reconstruction.
• Efficient level of detail (LOD): The representation naturally supports adaptive detail, where distant areas use fewer, larger Gaussians.
• Direct integration with GIS data: The explicit 3D points (Gaussian centers) can be directly tagged with geographic coordinates and other metadata.

The foundational implicit neural representation that 3D Gaussian Splatting was designed to surpass in speed. A NeRF uses a coordinate-based MLP to map a 3D location and viewing direction to a volume density and view-dependent color. It achieves photorealistic quality but is notoriously slow to train and render due to the need to query a deep network millions of times per ray.

• Core Difference: NeRF is an implicit, neural network-based representation, while 3DGS is an explicit, point-based representation.
• Rendering: NeRF uses volumetric ray marching through a continuous field; 3DGS uses tile-based rasterization of discrete primitives.

The essential computational framework that makes 3D Gaussian Splatting possible. Differentiable rendering formulates the image synthesis (rendering) pipeline as a function with computable gradients with respect to scene parameters (like Gaussian positions, colors, and opacities).

• Enables Optimization: Because the rasterizer is differentiable, gradients from a photometric loss (comparing rendered vs. real images) can flow backward to update each Gaussian's attributes.
• Key to Structure-from-Motion (SfM) Alignment: The initial point cloud for 3DGS often comes from SfM, which itself relies on differentiable principles to estimate camera poses.

An important precursor in the shift from implicit networks to explicit, optimizable primitives. Plenoxels are a sparse voxel grid representation where each voxel stores spherical harmonic coefficients for view-dependent color and a density value.

• Explicit & Differentiable: Like 3DGS, Plenoxels use an explicit data structure (a grid) and a differentiable volume renderer, enabling gradient-based optimization from images.
• Performance Bridge: Plenoxels demonstrated that high-quality novel view synthesis could be achieved without a large MLP, paving the way for even faster point-based methods like 3DGS.

The specific rasterization technique at the heart of 3D Gaussian Splatting. It projects 3D Gaussians onto the 2D image plane as 2D Gaussian splats (ellipses). The final pixel color is computed via alpha-blending of all splats overlapping that pixel, sorted by depth.

• Anisotropic: Gaussians are not simple spheres; they have a 3D covariance matrix defining their scale and rotation, allowing them to efficiently represent surface elements.
• Tile-Based Culling: For efficiency, the screen is divided into tiles. Only Gaussians whose 2D bounding boxes intersect a tile are considered for rasterization within that tile.

The mathematical basis used in 3D Gaussian Splatting to compactly represent view-dependent appearance. Each Gaussian stores a set of SH coefficients. During rendering, the viewing direction is evaluated against these coefficients to produce the final RGB color for that splat.

• Modeling Complex Effects: Low-order SH (e.g., 3 bands) capture basic diffuse shading, while higher orders (e.g., 4 bands as used in the original paper) can approximate specular highlights and more complex reflectance.
• Storage Efficiency: Storing a few dozen coefficients per Gaussian is far more efficient than storing a separate color for every possible viewpoint.

A contemporary, highly efficient neural scene representation that shares 3DGS's goal of real-time performance. Instant NGP uses a multi-resolution hash grid of learnable feature vectors, decoded by a tiny MLP.

• Encoding vs. Primitive: Instant NGP is an encoding technique (hash grid) for a compact neural network. 3DGS is a set of explicit primitives (Gaussians) with no decoding network.
• Trade-off: Instant NGP offers extremely fast training and compact size. 3DGS achieves even higher real-time rendering frame rates (> 100 FPS) and is more amenable to traditional GPU rasterization pipelines and editing.