Plenoxels

Unlike coordinate-based MLPs used in standard NeRFs, Plenoxels store scene information in an explicit, sparse voxel grid. Each grid vertex holds a set of learnable parameters, primarily spherical harmonic (SH) coefficients that model view-dependent color and a density value. This explicit structure eliminates the need for querying a large neural network for every 3D point, leading to significantly faster optimization and rendering times. The grid resolution can be adaptively refined during training to capture fine details.

To model how an object's appearance changes with the viewing angle (a key challenge in view synthesis), Plenoxels use low-degree spherical harmonics. Each voxel vertex stores SH coefficients (e.g., for degrees 0, 1, and 2) that compactly represent a function over the sphere of possible viewing directions. During rendering, the color for a specific ray direction is computed by evaluating this SH function. This provides a differentiable, memory-efficient way to capture non-Lambertian effects like specular highlights without the computational overhead of a per-ray MLP.

Plenoxels are rendered using the differentiable volume rendering integral, similar to NeRF. The key difference is in how properties are sampled:

• Density and SH coefficients at any continuous 3D point within the grid are computed via trilinear interpolation from the eight nearest grid vertices.
• This interpolation is fully differentiable, allowing gradients to flow back to the grid parameters during optimization.
• The process involves ray marching, accumulating color and opacity along each camera ray. This explicit, grid-based sampling is computationally efficient and avoids the expensive forward passes of a deep MLP.

A defining feature of Plenoxels is that the scene representation is optimized directly via gradient descent on the voxel grid parameters, without any neural network as the underlying function approximator. The optimization objective is a photometric loss (like Mean Squared Error) between the rendered novel views and the ground truth input images. Because the rendering pipeline is fully differentiable, gradients from this loss update the density values and spherical harmonic coefficients at each grid vertex. This often leads to convergence in minutes rather than the hours required for early NeRF models.

Plenoxels employ a sparse grid structure initialized from structure-from-motion points and are optimized with total variation (TV) regularization. This regularization encourages spatial smoothness and sparsity. During training, a pruning step removes voxels with opacity below a threshold, as they do not contribute significantly to the rendered scene. This results in a compact representation that focuses memory and computation on occupied regions of space, dramatically improving efficiency over a dense, uniform grid.

Plenoxels make a clear set of trade-offs compared to coordinate-MLP NeRFs:

• Speed: Training is 10-100x faster than the original NeRF due to the lack of a large MLP.
• Memory: The explicit grid can be memory-intensive for high-resolution scenes, though sparsity mitigates this.
• Quality: Achieves comparable, sometimes superior, visual quality on standard benchmarks, but may struggle with very high-frequency details that MLPs with positional encoding excel at.
• Editability: The explicit grid structure can be more intuitive for certain scene editing tasks compared to a 'black-box' neural network.

The foundational implicit neural representation that Plenoxels was designed to improve upon. A NeRF uses a coordinate-based MLP to map a 3D location and viewing direction to a volume density and color. This provides a continuous, memory-efficient scene model but is notoriously slow to train and render due to the need to query the MLP millions of times per image. Plenoxels emerged as a faster, explicit alternative that trades some compactness for computational speed.

A contemporary explicit, point-based representation that, like Plenoxels, enables real-time rendering. Instead of a voxel grid, it represents a scene with a cloud of anisotropic 3D Gaussians, each with attributes for position, covariance (scale/rotation), opacity, and spherical harmonic coefficients for view-dependent color. Its differentiable rasterization pipeline directly splats these primitives to the screen, achieving state-of-the-art quality and speed, positioning it as a successor to both NeRF and voxel-based methods like Plenoxels.

A hybrid representation that bridges explicit and implicit approaches. It uses a multi-resolution hash grid of learnable feature vectors as an explicit, efficient look-up table. A tiny coordinate-based MLP then decodes these features into color and density. This architecture delivers NeRF-quality results with training times of seconds to minutes, directly addressing the speed limitations of vanilla NeRF that also motivated Plenoxels, but with greater parameter efficiency.

The core optimization framework that makes techniques like Plenoxels and NeRF possible. It formulates the image synthesis (rendering) process as a differentiable function with respect to scene parameters (e.g., voxel densities, colors, Gaussian attributes). This allows the use of gradient descent to optimize these parameters by comparing rendered images to ground-truth input photos. The volume rendering integral is the specific differentiable equation used for volumetric representations like Plenoxels and NeRF.

A memory-efficient feature indexing technique central to Instant NGP and related to the grid structure of Plenoxels. It stores learnable feature vectors in a multi-resolution hash table, where spatial coordinates are hashed to table entries. This allows for a compact representation that gracefully handles hash collisions, enabling the modeling of fine details without a prohibitively large dense grid. It contrasts with Plenoxels' use of a dense, spatially-ordered voxel grid of spherical harmonic coefficients.

The mathematical foundation for rendering volumetric representations including Plenoxels, NeRF, and medical CT data. It models the color of a pixel as the integral of accumulated light and absorption along a camera ray passing through the volume. For Plenoxels, this is computed via numerical quadrature (ray marching), sampling points within each voxel. The integral's differentiability is what allows the voxel properties (density and spherical harmonic coefficients) to be optimized from 2D images.