Hierarchical Sampling

The process is divided into two distinct neural networks. The coarse network first samples points uniformly along each camera ray to produce an initial, low-resolution estimate of the scene's volume density. This density distribution is then used to guide the fine network, which performs importance sampling, concentrating its queries in regions where the coarse model predicts significant content (e.g., near surfaces). This architecture ensures the fine network wastes minimal computation on empty space.

The core mechanism enabling the fine stage is the construction of a Probability Density Function (PDF) from the coarse network's output. The normalized density values along a ray define this PDF. The fine network then samples a second set of points according to this distribution, a technique known as importance sampling. This biases the sampling towards regions that contribute most to the final rendered pixel color, as defined by the volume rendering integral. This is mathematically optimal for reducing variance in the gradient estimates during training.

By focusing samples where they matter most, hierarchical sampling directly reduces the variance of the Monte Carlo estimator used in the volume rendering integral. Lower variance leads to:

• Smoother, more stable gradients during backpropagation.
• Faster convergence of the neural radiance field.
• Higher fidelity reconstructions with the same total number of samples, or comparable quality with far fewer samples. This makes training a NeRF computationally feasible.

Hierarchical sampling is not a separate algorithm but is deeply integrated into the differentiable volume rendering pipeline. The final rendered color for a pixel is computed by accumulating outputs from both the coarse and fine samples along the ray. The loss function typically compares this composite rendering against the ground-truth pixel. This end-to-end design allows the coarse network to learn an effective proposal distribution specifically for optimizing the final photometric objective.

This method contrasts sharply with simpler alternatives:

• Uniform Sampling: Queries points at fixed intervals. Highly inefficient, as most samples are in empty space or homogeneous regions.
• Stratified Sampling: Jitters uniform samples within strata to reduce aliasing but doesn't adapt to scene content. Hierarchical sampling is adaptive; its sampling pattern changes dynamically based on the learned scene geometry, making it a form of learned sampling. This is key to NeRF's ability to model complex scenes with high detail near surfaces.

The hierarchical principle has been extended in subsequent NeRF research:

• Mip-NeRF: Replaces point sampling with cone tracing and uses integrated positional encoding to handle multi-scale anti-aliasing, but retains a hierarchical sampling structure.
• InstantNGP: Uses a multi-resolution hash grid for extremely fast feature lookup, but still employs a form of importance sampling to decide which regions of the spatial hash to query more densely. Thus, hierarchical sampling established a core design pattern for efficient neural scene representation.

Ray marching is the foundational volume rendering algorithm that hierarchical sampling optimizes. It approximates the light transport integral by:

• Casting a ray from the camera through each pixel.
• Sampling discrete 3D points at intervals along the ray.
• Querying the neural field for density and color at each point.
• Accumulating these values to produce a final pixel color. Hierarchical sampling improves upon naive, uniform ray marching by concentrating samples where they matter most.

Importance sampling is the statistical Monte Carlo technique that hierarchical sampling implements. Its goal is to reduce variance by sampling points with probability proportional to their expected contribution to the final rendered value. In NeRF:

• The coarse network provides a probability density function (PDF) along each ray.
• The fine network then samples additional points according to this PDF. This ensures computational effort is focused on regions of high density (likely surfaces), not empty space.

The volume rendering integral is the physical equation that NeRF approximates via ray marching and sampling. It computes the color C of a camera ray r as: C(r) = ∫ T(t) σ(r(t)) c(r(t), d) dt Where:

• σ is the volume density at point r(t).
• c is the radiance (color).
• T is the accumulated transmittance. Hierarchical sampling's two-stage process provides a better discrete approximation of this continuous integral than uniform sampling.

Coarse-to-fine optimization is a broader machine learning strategy where a simple model guides the training of a more complex one. In hierarchical sampling:

1. A coarse MLP is trained with uniformly sampled points to learn a low-resolution scene geometry.
2. This coarse model's output density informs where to sample finely.
3. A fine MLP (often the same network with a second forward pass) is then trained with these concentrated samples to capture high-frequency details. This strategy prevents the fine network from wasting capacity on empty space.

Mip-NeRF addresses a key limitation of the original hierarchical sampling strategy: it samples infinitesimal points, causing aliasing. Instead, Mip-NeRF:

• Models each ray as a 3D conical frustum.
• Uses Integrated Positional Encoding (IPE) to encode the volume of each sampled interval.
• Performs hierarchical sampling based on these intervals. This provides built-in anti-aliasing and more accurate multi-scale representation, making the sampling strategy more robust to varying camera distances.

Plenoxels represent an alternative, explicit approach to scene representation that eliminates the need for hierarchical sampling. Instead of a neural network, they use:

• A sparse voxel grid storing spherical harmonics coefficients for color and density.
• Differentiable volume rendering directly on this grid. Training uses a form of importance sampling but based on the explicit voxel densities, not a coarse network. This trade-off enables faster training but typically requires more memory than an implicit NeRF.