3D Gaussian Splatting

The core representation is a set of anisotropic 3D Gaussians. Each primitive is defined by:

• A 3D position (mean).
• A covariance matrix controlling scale and rotation (anisotropy).
• Opacity (alpha value).
• Spherical harmonics coefficients for view-dependent color. Unlike voxel grids or meshes, this representation is explicit and differentiable, allowing for efficient optimization and rendering. The anisotropy is key, allowing a single Gaussian to represent elongated surfaces like tree branches.

Rendering is performed via differentiable splatting. Each 3D Gaussian is projected onto the 2D image plane as a 2D Gaussian with a covariance matrix adjusted for perspective. The image is formed by alpha-blending these splats in depth order.

• The process is fully differentiable, enabling gradient-based optimization of all Gaussian parameters (position, covariance, color, opacity) from 2D image losses.
• This bridges computer graphics (rasterization/splatting) and computer vision (gradient descent), avoiding the expensive volumetric ray-marching used by Neural Radiance Fields (NeRF).

A primary advantage is real-time rendering at high resolutions (1080p+), achieving 100+ FPS on modern GPUs. This is enabled by:

• Fast GPU rasterization of splats using custom CUDA kernels.
• Tile-based culling to avoid processing Gaussians outside the view frustum.
• Depth sorting via a fast, approximate method. This performance makes it suitable for interactive applications like virtual reality, digital twins, and real-time previews, where NeRF-based methods are often too slow.

The 3D Gaussian scene representation is optimized from scratch using a set of posed input images. The process involves:

1. Initialization from a sparse point cloud (e.g., from Structure-from-Motion).
2. Adaptive Density Control: Gaussians are periodically cloned in under-reconstructed areas and pruned where they are transparent or redundant.
3. Loss Minimization: Parameters are optimized via stochastic gradient descent using a photometric loss (L1 + D-SSIM) between rendered and ground truth images. This yields a compact, high-quality scene representation in minutes to hours.

3D Gaussian Splatting uses an explicit scene representation, contrasting with the implicit representation of Neural Radiance Fields (NeRF).

• Explicit (Gaussians): Scene is a finite set of discrete, manipulable primitives. Enables fast rendering, easy editing, and direct storage.
• Implicit (NeRF): Scene is a continuous function (a neural network) queried at any 3D coordinate. Provides theoretically infinite resolution but is slower to render and query. This explicit nature facilitates tasks like scene editing, composition, and compression, though it may require more primitives for complex geometry.

This technique is transformative for generating and utilizing synthetic data:

• High-Fidelity Asset Creation: Rapid reconstruction of real-world objects for digital twins.
• Novel View Synthesis: Generating unlimited, perfectly labeled training views for computer vision models.
• Dynamic Scene Editing: Modifying reconstructed scenes (adding/removing objects) for data augmentation.
• Real-Time AR/VR: Enabling photorealistic, interactive spatial computing experiences. It serves as a bridge between photogrammetry, neural rendering, and real-time graphics pipelines.

A Neural Radiance Field (NeRF) is a foundational neural rendering technique that represents a 3D scene as a continuous volumetric function. It maps a 3D spatial coordinate and a 2D viewing direction to a volume density and a view-dependent RGB color. This implicit representation enables high-fidelity novel view synthesis from a sparse set of 2D images. While highly accurate, traditional NeRFs are computationally intensive for training and rendering, which motivated the development of faster alternatives like 3D Gaussian Splatting.

• Core Mechanism: Uses a multilayer perceptron (MLP) to encode the scene.
• Output: Density (sigma) and color (RGB) for any 3D point and viewing direction.
• Primary Use: Photorealistic static scene reconstruction and view synthesis.

Differentiable rendering is a framework where the classic graphics rendering pipeline is made differentiable with respect to its input parameters. This allows gradients to be computed from a loss function defined on the final 2D rendered pixels back to the 3D scene attributes (like geometry, texture, lighting). This gradient flow is essential for optimizing 3D representations from 2D images without explicit 3D supervision.

• Key Innovation: Enables optimization-through-rendering.
• Application in 3DGS: The splatting operation and associated blending are formulated as differentiable functions, allowing the positions, scales, rotations, and opacities of the 3D Gaussians to be optimized via gradient descent.
• Broader Impact: Critical for inverse graphics, single-view 3D reconstruction, and material estimation.

Novel view synthesis is the core computer vision and graphics task of generating a photorealistic image of a scene from a camera viewpoint that was not captured in the original input set of images. It is the primary benchmark application for techniques like NeRF and 3D Gaussian Splatting.

• Input: A set of images with known camera poses (from structure-from-motion).
• Output: A new image from an arbitrary, unseen camera position.
• Evaluation Metrics: Commonly uses Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and LPIPS (Learned Perceptual Image Patch Similarity) to compare synthesized views to ground-truth images.
• 3DGS Advantage: Achieves real-time rendering speeds for novel views, a significant advancement over prior neural rendering methods.

Point-based graphics is a computer graphics paradigm where a 3D scene is represented not by polygons (meshes) but by a cloud of discrete points, each with attributes like position, color, and normal. Rendering involves projecting these points (or small primitives like splats) onto the 2D image plane. 3D Gaussian Splatting is a sophisticated evolution of this concept.

• Historical Context: Early techniques like surface splatting used points as rendering primitives.
• 3DGS Contribution: Replaces simple points with anisotropic 3D Gaussians—ellipsoids that can be stretched and rotated—allowing for a more compact and accurate representation of surface geometry and view-dependent effects.
• Efficiency: Enables high-quality rendering without the topological constraints of meshes or the dense sampling of volumes.

Structure from Motion (SfM) is a photogrammetry technique that estimates the 3D structure of a scene (a sparse point cloud) and the camera poses (position and orientation) from a set of overlapping 2D images. It is a critical preprocessing step for 3D Gaussian Splatting and other neural rendering methods.

• Prerequisite for 3DGS: Provides the initial sparse point cloud used to initialize the positions of the 3D Gaussians.
• Output: Camera parameters (intrinsics and extrinsics) for every input image, which define the viewing transforms for rendering and optimization.
• Common Tools: COLMAP is the de facto standard open-source SfM system used in most 3DGS pipelines.

Tile-based rasterization is a graphics optimization technique where the screen is divided into small tiles (e.g., 16x16 pixels). Geometry (like triangles or, in 3DGS, Gaussian splats) is sorted and assigned to the tiles it overlaps before rendering begins. This drastically reduces memory bandwidth and improves cache coherence during the rendering pass.

• 3DGS Implementation: The 3DGS rendering pipeline uses a custom tile-based rasterizer that:
  1. Culls Gaussians outside the view frustum.
  2. Sorts remaining Gaussians by depth for each tile.
  3. Processes Gaussians in front-to-back order per tile, performing alpha blending.
• Performance Impact: This architecture is a key reason 3DGS achieves real-time frame rates, making it suitable for interactive applications.