3D Gaussian Splatting

The scene is represented by a collection of anisotropic 3D Gaussians, which are the fundamental primitives. Each Gaussian is defined by:

• A 3D center position (mean).
• A 3D covariance matrix, controlling its scale and rotation (shape).
• Opacity (alpha), controlling its contribution to the final pixel.
• Spherical harmonics coefficients, which encode view-dependent color.

Unlike isotropic spheres, the covariance matrix allows Gaussians to stretch and rotate, enabling efficient modeling of surface-like structures (e.g., flat leaves, thin rods) with far fewer primitives.

Rendering is performed by a custom differentiable tile-based rasterizer, which is key to real-time performance. The process is:

1. Projection & Sorting: 3D Gaussians are projected to 2D screen space. A fast, tile-based renderer sorts them per 16x16 pixel tile, ensuring only relevant Gaussians are processed.
2. Alpha Blending: Within each tile, Gaussians are sorted by depth and blended front-to-back using alpha compositing.
3. Differentiability: The entire rasterization pipeline is designed to be differentiable, allowing gradients to flow back from the 2D image loss to update the 3D Gaussian parameters (position, covariance, color, opacity) during optimization.

The representation starts sparse and becomes denser through an adaptive density control process during training. This is a core optimization mechanism:

• Clone: Gaussians in areas with large positional gradient (under-reconstruction) are cloned to increase local detail.
• Split: Large Gaussians (oversized) are split into smaller ones to better capture fine-grained geometry.
• Prune: Gaussians with very low opacity (transparent) are periodically removed.

This process dynamically grows the set of Gaussians from an initial sparse point cloud (from Structure-from-Motion), creating an efficient, detail-adaptive scene representation without manual intervention.

A primary advantage over NeRF is real-time rendering at high resolutions (e.g., 1080p at > 100 FPS). This is achieved because:

• Rasterization vs. Ray Marching: It uses traditional graphics rasterization pipelines, which are massively parallelized on GPUs, instead of the sequential ray marching used by NeRF.
• Explicit Primitives: Gaussians are explicit, view-independent primitives. Their screen-space projection and blending is a highly optimized operation.
• Level of Detail (LOD): The Gaussian representation can be simplified for distant objects, though the core method typically renders all primitives. This enables interactive applications like VR and AR where low latency is critical.

The set of 3D Gaussians forms an explicit scene representation. Each Gaussian is a discrete, manipulable entity, which enables practical editing operations that are challenging with implicit representations like NeRF:

• Selective Pruning: Objects can be removed by deleting their constituent Gaussians.
• Geometry Manipulation: Gaussians can be translated, rotated, or scaled by adjusting their mean and covariance.
• Appearance Editing: Color can be modified by adjusting the spherical harmonics coefficients.
• Compositing: Scenes can be combined by merging Gaussian sets. This explicit nature bridges neural rendering with traditional computer graphics pipelines.

The model is optimized from posed images using a photometric loss function, similar to NeRF, but with a key difference in the rendering mechanism. The standard loss is: L = (1 - λ) * L1 + λ * L_D-SSIM

• L1 Loss: Measures absolute pixel-wise difference between rendered and ground truth images.
• D-SSIM Loss: The structural dissimilarity index measure (D-SSIM) accounts for perceptual quality and encourages sharper textures.
• λ: A balancing weight (typically ~0.2).

Gradients from this loss update all Gaussian parameters via the differentiable rasterizer. No 3D ground truth (like meshes) is required, only multi-view 2D images and their camera poses.

3D Gaussian Splatting excels at generating photorealistic images from arbitrary, unseen camera angles in real-time. This is achieved by rasterizing millions of anisotropic 3D Gaussians directly onto the 2D image plane using a fast, tile-based renderer. Unlike Neural Radiance Fields (NeRF) which require slow ray marching, this method enables interactive frame rates (> 100 FPS) on consumer GPUs, making it ideal for:

• Virtual and Augmented Reality experiences where low latency is critical.
• Free-viewpoint video for sports broadcasting and entertainment.
• Interactive 3D scene exploration from sparse photo collections.

The technique provides an explicit, editable 3D scene representation suitable for creating digital twins. The scene is composed of a set of 3D Gaussians, each with attributes like position, covariance (scale/rotation), color (via spherical harmonics), and opacity. This representation is:

• Compact and Efficient: Often requires only 100-500 MB per scene, compared to gigabytes for dense neural networks or point clouds.
• Explicit and Editable: Individual Gaussians can be manipulated, removed, or duplicated, enabling scene editing and composition.
• Fast to Optimize: Training (via differentiable rendering and photometric loss) typically converges in minutes to tens of minutes, far faster than many NeRF variants.

Extensions to 3D Gaussian Splatting enable the modeling of non-rigid, moving scenes. By treating Gaussian attributes as functions of time or by learning deformation fields, the method can represent:

• Dynamic Objects: People, animals, and vehicles in motion.
• Deforming Surfaces: Cloth, fluids, or facial expressions.
• Time-varying Appearances: Changes in lighting or material properties. This is crucial for applications in volumetric capture for filmmaking, telepresence, and creating dynamic assets for simulations and games.

The real-time capability and explicit geometry of Gaussian Splatting make it valuable for robotic perception and planning.

• Sim-to-Real Transfer: High-fidelity synthetic environments can be generated quickly for training reinforcement learning agents.
• Scene Understanding: The explicit 3D Gaussians can be segmented or classified to identify objects and free space for navigation.
• Dense Mapping: Robots can build dense, photorealistic maps of their environment in real-time, useful for simultaneous localization and mapping (SLAM) and inspection tasks.

In AEC, Gaussian Splatting enables rapid visualization and analysis from sparse data.

• Site Progress Monitoring: Creating up-to-date 3D models from daily drone or camera feeds for comparison against BIM (Building Information Modeling) plans.
• Virtual Walkthroughs: Generating immersive, interactive tours of construction sites or existing buildings from simple photo scans.
• Asset Management: Creating searchable, photorealistic inventories of complex facilities like factories or plants.

The pipeline offers a fast workflow for generating high-quality 3D assets from real-world objects.

• Asset Generation: Artists can quickly capture real-world objects (e.g., sculptures, props) and convert them into usable, view-consistent 3D representations.
• Environment Building: Entire scenes can be reconstructed from video for use as background plates or fully navigable environments in games and virtual production.
• Hybrid Rendering: Gaussian Splats can be integrated into traditional rasterization or ray-tracing pipelines as efficient, detailed neural assets, blending learned and conventional graphics.

Neural Radiance Fields (NeRF) is the foundational deep learning technique that 3D Gaussian Splatting builds upon and contrasts with. It represents a 3D scene as a continuous volumetric function, parameterized by a multilayer perceptron (MLP). This function maps a 3D spatial coordinate and a 2D viewing direction to a volume density and a view-dependent RGB color. Rendering an image requires computationally intensive ray marching, where hundreds of samples along each camera ray are queried through the MLP and composited. While NeRFs produce extremely high-fidelity, photorealistic novel views, their primary limitation is slow rendering speed (seconds per frame), making them unsuitable for real-time applications without significant acceleration structures.

Differentiable rendering is the critical enabling framework that allows techniques like 3D Gaussian Splatting and NeRF to be optimized from 2D images. It makes the graphics rendering pipeline—the process of generating a 2D image from 3D scene parameters—mathematically differentiable. This means gradients of pixel colors with respect to scene attributes (like Gaussian positions, colors, or neural network weights) can be computed and propagated backward. Key aspects include:

• Alpha Blending: The core compositing operation in rasterization-based splatting, which must be made differentiable.
• Gradient Flow: Enables the use of gradient descent to adjust 3D representations to minimize a photometric loss between rendered and ground truth images.
• Bridge to Graphics: It connects deep learning optimization with traditional computer graphics primitives and pipelines.

Novel view synthesis is the core computer vision task that 3D Gaussian Splatting is designed to solve. The goal is to generate photorealistic images of a static or dynamic scene from arbitrary camera viewpoints that were not present in the original set of input images. It is a fundamental capability for:

• Virtual and Augmented Reality: Allowing users to look around a captured environment.
• Cinematography: Creating new camera angles from a limited set of shots.
• Spatial Computing: Building 3D understanding from 2D imagery. 3D Gaussian Splatting addresses this by providing a explicit, rasterizable 3D representation that can be projected and blended in real-time, offering a significant speed advantage over prior neural implicit methods for this specific task.

Instant Neural Graphics Primitives (Instant NGP) is a contemporaneous acceleration framework for NeRF that shares 3D Gaussian Splatting's goal of real-time performance but uses a different technical approach. Instead of using explicit Gaussians, it retains an implicit neural representation but dramatically speeds it up using:

• Multi-Resolution Hash Encoding: A compact, trainable data structure that maps spatial coordinates to feature vectors via a hierarchy of hash tables, enabling the MLP to be small and fast.
• Fully Fused CUDA Kernels: Highly optimized GPU implementations. While both achieve real-time rendering, 3D Gaussian Splatting uses a rasterization-based pipeline, whereas Instant NGP uses a highly accelerated ray marching-based pipeline. They represent two parallel paths toward solving the speed limitation of original NeRF.

Point-Based Rendering is the classical computer graphics lineage from which 3D Gaussian Splatting directly descends. It represents geometry as a set of discrete surface points (or splats) rather than polygons. Splatting is the rasterization technique that projects these 3D points onto the 2D image plane, typically as oriented disks or ellipses with attributes like color and opacity, which are then blended. 3D Gaussian Splatting's key innovations are:

• Using anisotropic 3D Gaussians as the splat primitive, allowing them to adapt to surface geometry.
• Making the entire pipeline differentiable and optimizable from images.
• Introducing a density-based adaptive control mechanism for creating and pruning Gaussians during training. This modernizes a decades-old idea with learnable, high-quality parameters.

Mesh Extraction is the process of converting an implicit or point-based 3D representation into an explicit, polygonal mesh (a set of vertices and faces). This is a common post-processing step for many neural 3D representations to enable use in standard game engines, CAD software, or 3D printing. Techniques like Marching Cubes are used to extract a mesh from a Signed Distance Function (SDF). A key differentiator for 3D Gaussian Splatting is that it is an explicit representation from the start—the Gaussians are discrete, optimizable entities. However, they are not a connected surface mesh. Converting Gaussians to a usable mesh remains a non-trivial research problem, whereas a trained NeRF or SDF can be more directly meshed, albeit with potential artifacts.