3D Gaussian Splatting

The core of the representation is a set of 3D Gaussian primitives. Unlike isotropic (spherical) Gaussians, these are anisotropic, meaning their shape can stretch and rotate in 3D space. Each Gaussian is defined by:

• A mean (μ): Its 3D position in world coordinates.
• A covariance matrix (Σ): A 3x3 matrix that controls its anisotropic shape (scale and rotation).
• Opacity (α): Controls the contribution of the Gaussian to the final pixel.
• Spherical Harmonics (SH) coefficients: Store view-dependent color information, allowing materials to exhibit effects like specular highlights.

Rendering is performed by a custom differentiable rasterizer that projects millions of Gaussians onto the 2D image plane. The process is optimized for real-time performance using a tile-based approach:

• The screen is divided into tiles (e.g., 16x16 pixels).
• Gaussians are sorted and assigned to tiles they overlap using fast, GPU-friendly culling.
• Within each tile, Gaussives are alpha-blended in sorted depth order.
• The rasterizer is fully differentiable, enabling end-to-end optimization of all Gaussian parameters (position, covariance, color, opacity) from posed 2D images via gradient descent.

The set of Gaussians is not static; it is dynamically optimized during training through adaptive density control. This process manages the primitive count and placement:

• Cloning: Gaussians in areas of high positional gradient (under-reconstruction) are cloned to increase detail.
• Pruning: Gaussians with very low opacity are periodically removed to optimize performance.
• This allows the representation to start from a sparse point cloud (e.g., from Structure from Motion) and adaptively increase its density in complex regions (e.g., textured surfaces) while remaining sparse in empty or uniform areas.

A primary advantage over Neural Radiance Fields (NeRF) is real-time inference. Once trained, rendering a novel view involves:

• A fast forward pass through the rasterizer, with performance scaling linearly with the number of splats.
• Achieving > 100 FPS at 1080p resolution on modern GPUs.
• The rendering cost is largely independent of scene complexity, as it depends on the number of Gaussians projected per pixel, not a uniform sampling of a volumetric field.
• This makes it suitable for interactive applications like VR, AR, and real-time simulation.

3D Gaussian Splatting is an explicit scene representation, in contrast to the implicit function of a NeRF. This has key implications:

• The Gaussians are discrete, manipulable entities. Individual primitives can be selected, edited, or animated.
• The representation is natively compatible with traditional graphics pipelines and can be exported for use in standard 3D software (with some conversion).
• Storage is efficient; a complex scene can be represented with ~500k to 3 million Gaussians, requiring only tens to hundreds of megabytes.

While both aim for photorealistic view synthesis, their mechanisms differ fundamentally:

• NeRF: Uses a multilayer perceptron (MLP) to encode a scene as a continuous volumetric field (density and color). Rendering requires expensive ray marching and MLP queries, making it slow.
• 3D Gaussian Splatting: Uses an explicit set of primitives. Rendering is a fast splatting/rasterization process.
• Training Speed: Gaussian Splatting often trains significantly faster (minutes to an hour) compared to NeRF (hours to days).
• Artifacts: NeRFs can suffer from blurriness; Gaussians can exhibit 'blobbiness' or popping if not well-optimized, but generally produce sharp textures.

The scene is modeled by thousands to millions of anisotropic 3D Gaussians. Each Gaussian is defined by:

• A mean (μ): Its 3D position in space.
• A covariance matrix (Σ): Defines its ellipsoidal shape and orientation.
• Opacity (α): Controls its contribution to the final pixel.
• Spherical Harmonics (SH) coefficients: Encode view-dependent color (radiance). This explicit, differentiable representation is optimized from posed input images.

Rendering is performed via tile-based rasterization. For a target camera view:

1. Projection: 3D Gaussians are projected to 2D, becoming 2D Gaussians on the image plane.
2. Tile Culling: Gaussians are sorted into screen-space tiles for efficient parallel processing.
3. Alpha Blending: Pixels are computed by blending overlapping 2D Gaussians in depth order using the classic volume rendering equation. This entire pipeline is differentiable, enabling gradient-based optimization from image loss.

The Gaussian set is optimized via stochastic gradient descent to minimize the difference between rendered and ground truth images. A key innovation is adaptive density control:

• Cloning: Gaussians with large positional gradients (under-reconstruction) are split.
• Pruning: Gaussians with very low opacity are removed.
• Resizing: Covariances are adjusted. This dynamic process starts from a sparse point cloud (from Structure from Motion) and grows a detailed representation only where needed.

The primary use case is generating photorealistic images from arbitrary, unseen camera positions in real-time (often > 100 FPS). This surpasses prior neural methods like Neural Radiance Fields (NeRF) in speed by orders of magnitude. Applications include:

• Virtual and Augmented Reality: Walkthroughs of captured spaces.
• Cinematography: Free-viewpoint video.
• Robotics Simulation: Fast rendering for training and planning.
• Digital Archival: Interactive exploration of cultural heritage sites.

Compared to NeRF, 3D Gaussian Splatting offers distinct benefits:

• Speed: Explicit splatting enables real-time rendering without costly volumetric sampling.
• Explicit Storage: The Gaussian set is a compact, editable, and portable asset.
• High-Fidelity Details: Can capture fine structures like hair and grass more effectively.
• Natural Anti-Aliasing: The Gaussian formulation inherently supports level-of-detail. The trade-off is potentially larger storage size and less inherent smoothness than a continuous implicit function.

Neural Radiance Fields (NeRF) is a foundational deep learning technique for novel view synthesis. It represents a scene as a continuous volumetric function—typically a multilayer perceptron—that maps a 3D location and viewing direction to an RGB color and volume density. Training involves minimizing the difference between rendered and actual 2D images via volume rendering. While producing photorealistic results, traditional NeRFs are notoriously slow to train and render, which 3D Gaussian Splatting was designed to address.

• Core Mechanism: A neural network encodes the scene's radiance field.
• Output: Color and density for any 3D point and view direction.
• Contrast with 3DGS: NeRFs are implicit, continuous representations; 3DGS uses an explicit set of discrete, optimizable primitives for real-time performance.

A point cloud is a fundamental, unstructured 3D data format consisting of a set of discrete data points in a coordinate system (X, Y, Z), often with additional attributes like color or intensity. They are the raw output of sensors like LiDAR, RGB-D cameras, and photogrammetry. 3D Gaussian Splatting can be initialized from a sparse point cloud generated by Structure-from-Motion (SfM). The key evolution in 3DGS is that these points become anisotropic 3D Gaussians—optimizable primitives with covariance, opacity, and spherical harmonics for view-dependent color—rather than remaining static, unrenderable points.

• Data Source: Direct sensor output or reconstructed from images.
• Role in 3DGS: Provides the initial spatial positions for Gaussians.
• Limitation: Raw point clouds lack a native rendering mechanism for photorealistic novel views.

Structure from Motion (SfM) is a photogrammetry technique that estimates a sparse 3D point cloud and the camera poses (positions and orientations) from a collection of unordered 2D images of a static scene. It is a critical preprocessing step for 3D Gaussian Splatting. The SfM pipeline (e.g., using COLMAP) identifies matching features across images, triangulates their 3D positions, and refines everything via bundle adjustment. The output sparse point cloud and camera parameters serve as the initial geometry and supervision signal for training the 3D Gaussians.

• Primary Function: Recovers 3D structure and camera motion from 2D images.
• Output for 3DGS: Initial Gaussian positions and camera calibration data.
• Key Algorithm: Bundle adjustment for joint optimization of points and poses.

Multi-View Stereo (MVS) and Surface Reconstruction are techniques for creating dense, continuous 3D models from images. MVS takes the camera poses from SfM and generates a dense point cloud or depth maps for each view. Surface Reconstruction algorithms (e.g., Poisson reconstruction, Screened Poisson) then 'skin' these points to create a polygon mesh (a watertight surface). 3D Gaussian Splatting offers an alternative paradigm: instead of an explicit mesh, it creates a differentiable, renderable representation optimized directly for view synthesis, bypassing the often-fragile meshing step and excelling at representing complex appearances like fur and foliage.

• MVS Goal: Create dense geometry from posed images.
• Surface Output: A polygon mesh (e.g., .obj, .ply file).
• 3DGS Difference: Avoids explicit surface inference; optimizes for rendering quality directly.

Volumetric rendering is the core graphics technique used by both NeRF and 3D Gaussian Splatting to generate a 2D image from a 3D representation. It works by simulating the accumulation of light along camera rays. For 3DGS, rendering involves:

1. Projecting 3D Gaussians to 2D.
2. Sorting them by depth along each pixel's ray.
3. Blending their colors and opacities using alpha compositing.

The differentiable splatting operation and this blending process allow gradients to flow back during training, enabling the optimization of Gaussian parameters (position, covariance, color, opacity) from only 2D image supervision.

• Core Principle: Integrate color/opacity along viewing rays.
• 3DGS Implementation: Differentiable alpha blending of sorted 2D splats.
• Key Advantage: Enables end-to-end optimization from images.

A Signed Distance Function (SDF) is an alternative implicit 3D representation where the value at any 3D point is the distance to the nearest surface, with sign indicating inside (negative) or outside (positive). Neural implicit surfaces (like NeuS, VolSDF) use a neural network to encode this SDF, enabling high-quality surface reconstruction. Compared to 3D Gaussian Splatting:

• SDFs/Implicit Surfaces: Excel at extracting high-fidelity, watertight meshes via ray marching or the Marching Cubes algorithm. They represent a precise boundary.
• 3D Gaussians: Excel at real-time, photorealistic rendering of complex appearance and unbounded scenes. They represent a volumetric radiance field optimized for rendering speed, not a pure surface. The choice depends on the end goal: precise geometry (SDF) vs. real-time view synthesis (3DGS).