Plenoxels

Unlike a Neural Radiance Field (NeRF) which uses an implicit neural representation, Plenoxels store scene properties directly in an explicit, sparse voxel grid. Each grid vertex holds learnable coefficients for spherical harmonics (for view-dependent color) and a density value. This structure eliminates the need for a large Multi-Layer Perceptron (MLP) during inference, making rendering a direct lookup and interpolation process.

To model view-dependent effects like specular highlights, Plenoxels use low-degree spherical harmonics (SH) coefficients stored at each voxel. The color (c) for a given viewing direction is computed by evaluating the SH basis functions. This provides a compact, differentiable representation of the 5D plenoptic function (3D position + 2D direction) without requiring the viewing direction as a network input at every query.

Plenoxels are optimized via differentiable volume rendering. Rays are cast from the camera, and properties from sampled points are trilinearly interpolated from the eight nearest voxel vertices. The final pixel color is computed using the volume rendering equation, integrating density and radiance along the ray. Crucially, this entire pipeline is differentiable, allowing gradients to flow back to update the voxel grid parameters via stochastic gradient descent.

The voxel grid parameters (SH coefficients and density) are optimized directly. The process:

• Renders images from training camera poses.
• Computes a photometric loss (e.g., MSE) against ground truth images.
• Uses the gradient of this loss to update the explicit voxel parameters.
• Employs total variation (TV) regularization on the grid to encourage smoothness and reduce artifacts, acting as a prior for natural scenes.

The method starts with a dense voxel grid but uses pruning to achieve sparsity. Voxels with near-zero opacity (density) are identified and removed during optimization. This creates an adaptive, sparse voxel grid that concentrates representation in occupied regions of space, dramatically improving memory efficiency compared to a dense 3D grid.

The primary advantage over NeRF is test-time speed. Once optimized, rendering a novel view requires no neural network evaluations. It involves:

• Ray-voxel intersection to find the sparse set of intersected voxels.
• Trilinear interpolation of stored parameters.
• Spherical harmonics evaluation for color.
• Volume rendering integration. This results in rendering times orders of magnitude faster than standard NeRF, though still slower than traditional triangle rasterization.

Neural Radiance Fields (NeRF) is the foundational technique that Plenoxels builds upon. It represents a 3D scene as a continuous volumetric function, parameterized by a multi-layer perceptron (MLP). This network maps a 3D coordinate and 2D viewing direction to a volume density and view-dependent radiance. The primary innovation is using differentiable volume rendering to train this network from a set of 2D images, enabling photorealistic novel view synthesis. Plenoxels replaces the neural network with an explicit, optimized voxel grid for faster rendering.

This is the core distinction defining Plenoxels' architecture.

• Explicit Representations (like Plenoxels) store scene properties (e.g., density, color coefficients) in a discrete, structured data structure such as a voxel grid, point cloud, or mesh. They enable fast, non-iterative rendering at test time but can require significant memory for high resolution.
• Implicit Representations (like standard NeRF) encode the scene within the weights of a neural network. They are continuous and memory-efficient for complex geometry but require running the network for every query, making rendering slower. Plenoxels chooses explicitness for inference speed.

A voxel grid is the fundamental 3D data structure used by Plenoxels. Think of it as a 3D pixel (volume pixel) grid. In Plenoxels, each vertex of this grid stores learned coefficients:

• Spherical Harmonics (SH) Coefficients: Model the view-dependent radiance (color) from any direction.
• Density (σ): Defines the geometry and opacity at that location. During optimization, these values are directly adjusted via gradient descent. For rendering, properties at any point within the grid are computed via trilinear interpolation of the nearest eight vertices, making the representation differentiable.

Spherical Harmonics (SH) are a mathematical system of orthogonal functions defined on the surface of a sphere. Plenoxels uses them to compactly represent view-dependent appearance.

• Instead of storing a separate color for every possible viewing direction (which is infinite), Plenoxels stores a small set of SH coefficients (e.g., 9 for 3 color channels) at each voxel vertex.
• During rendering, the 2D viewing direction is converted to spherical coordinates and evaluated against the SH basis functions, weighted by the stored coefficients, to produce the final RGB color.
• This provides a smooth, continuous approximation of how color changes with viewpoint, crucial for modeling effects like specular highlights.

This is the core algorithm that enables the training of both NeRFs and Plenoxels from 2D images. The process is:

1. Ray Casting: For each pixel, a ray is cast from the camera into the scene.
2. Point Sampling: Points are sampled along the ray.
3. Property Query: At each sample point, density and color are queried (from the voxel grid in Plenoxels).
4. Alpha Compositing: Colors and densities are accumulated along the ray using the volume rendering equation to produce a final pixel color. Because every step (especially interpolation and compositing) is differentiable, gradients of the photometric loss (difference between rendered and real image) can flow back to update the voxel grid's density and SH coefficients.

InstantNGP is a contemporary acceleration technique for NeRF that shares Plenoxels' goal of fast training and rendering but uses a different hybrid approach.

• It employs a multi-resolution hash grid as an explicit, learnable feature encoding.
• A tiny MLP then decodes these features into density and color.
• Key Comparison: While both use explicit structures (voxel grid vs. hash grid), InstantNGP retains a small neural network decoder. Plenoxels is entirely non-neural at inference—the grid stores the final renderable coefficients directly. InstantNGP typically trains faster, but Plenoxels can achieve very high-quality, network-free rendering.