Plenoxels

Unlike an implicit neural representation like a standard NeRF, Plenoxels store scene data in an explicit, sparse 3D grid of voxels. Each voxel contains learnable parameters for spherical harmonics coefficients (modeling view-dependent color) and a density value. This structure eliminates the need for a large MLP to decode spatial coordinates, making optimization and rendering significantly faster.

• Direct Parameterization: Scene properties are directly attached to grid locations.
• Sparsity: Only voxels occupied by the scene are stored and optimized, saving memory.
• Interpolation: Properties for any 3D point are computed via trilinear interpolation of the eight nearest voxels.

Plenoxels are optimized via differentiable volumetric rendering using the classic volume rendering equation. A ray is cast through the voxel grid, and color and density values are sampled and composited along its path. Because the rendering process is fully differentiable with respect to the voxel parameters, gradients can flow backward to update the spherical harmonics and density values.

• Ray Marching: The scene is rendered by accumulating samples along camera rays.
• Gradient Flow: The mean squared error (MSE) between rendered and ground-truth images provides the loss signal to update all voxel parameters via stochastic gradient descent.
• No Neural Network: The differentiability is inherent to the rendering math, not a neural network, simplifying the pipeline.

The Plenoxel grid is trained solely through gradient-based optimization (typically using the Adam optimizer) on a set of posed 2D images. There is no traditional "training" of neural network weights; instead, the numerical values in the voxel grid itself are the parameters being optimized.

• Fast Convergence: The explicit representation often converges in minutes, compared to hours for early NeRFs.
• Total Variation (TV) Regularization: A regularization loss is applied to encourage spatial smoothness in the voxel grid, reducing noise and artifacts.
• Progressive Grid Refinement: Training often starts on a coarse grid (e.g., 128³) and is upsampled to finer resolutions (e.g., 512³) for detail.

To model complex materials and lighting, each voxel stores coefficients for low-degree spherical harmonics (SH). Spherical harmonics are a set of basis functions defined on a sphere. When a view direction is provided, the SH basis functions are evaluated and combined using the stored coefficients to produce a view-dependent RGB color for that voxel.

• Compact Representation: A few SH coefficients (e.g., 2nd or 3rd order) can accurately represent non-Lambertian effects like specular highlights.
• Directional Query: The final color c at a point is a function: c = SH(direction, coefficients).

The primary advantage of Plenoxels is computational efficiency. By replacing a large MLP with a sparse grid and trilinear interpolation, both optimization and rendering are accelerated.

• Fast Training: Initial optimization can complete in 5-25 minutes on a single GPU.
• Fast Inference: Rendering a novel view requires a single forward pass of the (fixed) voxel grid through the volumetric renderer, which can be highly optimized.
• Memory Mapped: The dense grid structure is amenable to efficient GPU memory access patterns and caching.

Plenoxels are a direct, discretized representation of the plenoptic function, a core concept in light field and image-based rendering. The plenoptic function P(x, y, z, θ, φ, λ, t) describes the intensity of light of wavelength λ at every 3D point (x,y,z), in every direction (θ,φ), at every time t.

• Simplified Model: Plenoxels model a static (t is constant), monochromatic/trichromatic (λ as RGB) slice of this function.
• 5D Representation: For a static scene, it becomes a 5D function: P(x, y, z, θ, φ).
• Voxel Storage: The voxel grid stores a local approximation of this 5D function at each grid point via its density and spherical harmonics coefficients.

A Neural Radiance Field (NeRF) is an implicit scene representation that uses a multilayer perceptron (MLP) to map a 3D coordinate and viewing direction directly to a volume density and view-dependent RGB color. Unlike the explicit grid of Plenoxels, a NeRF is a continuous function approximated by a neural network, requiring significant computation for both training and rendering via ray marching. Its key innovation is enabling photorealistic novel view synthesis from sparse 2D images.

Instant Neural Graphics Primitives (Instant NGP) is a framework that dramatically accelerates NeRF training and rendering. It replaces the large MLP with a small network fed by a multi-resolution hash grid encoding of spatial features. This hybrid approach—using an explicit, learnable data structure for fast feature lookup—enables real-time inference. It represents a key evolution from pure implicit models (NeRF) toward the explicit-neural hybrid paradigm that also includes Plenoxels.

An explicit-neural hybrid is a class of scene representations that strategically combines an explicit data structure (e.g., a voxel grid, hash table, or set of planes) with a compact neural network. The explicit structure stores coarse or multi-scale features, while the small network decodes these into final properties like color and density. This architecture, used by both Plenoxels and Instant NGP, is designed to balance the fast optimization and rendering of explicit methods with the high fidelity and compactness of neural representations.

Tri-plane features are a compact explicit-neural representation where learnable feature vectors are stored on three orthogonal, axis-aligned 2D feature planes (XY, XZ, and YZ). To query a 3D point, features are bilinearly interpolated from each plane, concatenated, and decoded by a small MLP. This approach, popularized by EG3D, provides a memory-efficient alternative to dense 3D grids (like the initial Plenoxel structure) and is highly effective for representing bounded objects or scenes.

Differentiable rendering is a framework that allows gradients to flow from a rendered 2D image back to the underlying 3D scene parameters (like geometry, materials, and lighting). This gradient flow enables the optimization of scene representations from collections of 2D images. Both Plenoxels and NeRFs rely on a differentiable rendering process—specifically, differentiable volume rendering—to adjust voxel properties or network weights to minimize photometric loss between rendered and ground truth images.

The plenoptic function is a theoretical complete representation of all light rays in a scene, defined as the intensity of light seen from every 3D spatial position (x,y,z), in every direction (θ,φ), for every wavelength (λ), and at every time (t). The core challenge of novel view synthesis is to reconstruct an approximation of this function. Plenoxels directly model a discretized version of the static plenoptic function (7D, reduced to 5D by omitting time and often modeling RGB), storing spherical harmonic coefficients per voxel to represent directional appearance.