Ray Marching

Unlike ray tracing which finds exact geometric intersections, ray marching progresses along a ray in discrete steps. At each step, it queries a Signed Distance Function (SDF) or density field to determine the distance to the nearest surface. The ray then 'marches' forward by that distance, ensuring efficient progression through empty space and accurate convergence on surfaces. This is the foundational algorithm for sphere tracing, a common method for rendering implicit surfaces.

• Core Mechanism: ray_position += step_size * direction
• Key Function: float sdf = sceneSDF(current_position);
• Advantage: Naturally handles complex, procedurally defined geometry where analytic intersections are unknown.

In NeRF, ray marching is used to approximate the volumetric rendering equation. The final pixel color (C(\mathbf{r})) for a ray (\mathbf{r}) is computed by numerically integrating accumulated color and transparency along its path:

(C(\mathbf{r}) = \sum_{i=1}^N T_i , (1 - \exp(-\sigma_i \delta_i)) , \mathbf{c}_i)

Where:

• (\sigma_i) is the volume density (opacity) at sample (i).
• (\mathbf{c}_i) is the radiance (color) at sample (i).
• (\delta_i) is the distance between samples.
• (T_i = \exp(-\sum_{j=1}^{i-1} \sigma_j \delta_j)) is the transmittance (how much light reaches sample (i)).

This integral is what allows NeRF to model semi-transparent effects, fog, and complex light scattering.

Naive uniform sampling along a ray is computationally wasteful. Hierarchical sampling, as introduced in the original NeRF paper, uses a two-stage process:

1. A coarse network is queried at uniformly sampled locations to produce a preliminary density distribution.
2. This distribution informs a fine network, which performs importance sampling—concentrating more samples in regions likely to contain visible surfaces (high density).

This strategy drastically improves sample efficiency, allocating computation where it matters most for final image quality. Advanced methods like inverse transform sampling are used to draw samples proportional to the predicted density distribution.

A critical feature of ray marching in NeRF is that the entire process is differentiable. The gradients of the photometric loss (difference between rendered and real pixel colors) can flow backward through the rendering integral, through the sampled densities and colors, and into the parameters of the Multi-Layer Perceptron (MLP) that defines the scene. This enables:

• Gradient-based optimization of scene geometry and appearance from only 2D images.
• Joint optimization of camera poses alongside the neural field, as in Bundle-Adjusting NeRF (BARF).
• Training of generative 3D models via guidance from 2D diffusion models using Score Distillation Sampling (SDS).

The differentiability chain is: Pixel Loss → Rendered Color → Sample Weights (Alpha compositing) → Neural Network Outputs (σ, c) → MLP Weights.

Pure neural ray marching is computationally intensive. Modern implementations use hybrid representations and acceleration structures to achieve interactive speeds:

• Multi-Resolution Hash Grids (InstantNGP): Stores features in a spatially hashed grid, allowing a tiny MLP to look up local features instead of memorizing the entire scene, enabling training in seconds.
• Explicit Voxel Grids (Plenoxels): Store density and spherical harmonic coefficients directly in a sparse voxel grid, eliminating the MLP at inference for faster rendering.
• Tensor Factorization (TensorRF): Decomposes the 4D radiance field into compact low-rank tensor components, reducing model size and speeding up queries.
• Occupancy Grids / Bounding Volume Hierarchies (BVH): Skip large empty regions of space by testing against a coarse, pre-computed occupancy map before querying the neural network.

Standard ray marching treats rays as infinitesimally thin lines, causing aliasing (jaggies) when rendering at different resolutions. Mip-NeRF addresses this by modeling each pixel as a cone (or conical frustum). Instead of sampling points, it integrates the scene representation over the volume of the cone. This is achieved using:

• Integrated Positional Encoding (IPE): Computes the expected (mean and variance) frequency encoding over the conical region, rather than at a single point.
• Conical Frustum Sampling: Samples are drawn within the conical volume, not along a single line.

This results in anti-aliased renders that are stable across multiple scales and viewing distances, mimicking the behavior of a physical camera's finite aperture.

Ray marching in NeRF is a numerical approximation of the volume rendering integral. For each pixel, a ray r(t) = o + td is cast from the camera center o in direction d. The expected color C(r) of the ray is computed by:

• Sampling discrete points t_i along the ray's near/far bounds.
• Querying the NeRF MLP at each point for density σ and color c.
• Accumulating these values using alpha compositing: C(r) = Σ_i T_i α_i c_i, where T_i = exp(-Σ_{j=1}^{i-1} σ_j δ_j) is transmittance and α_i = 1 - exp(-σ_i δ_i) is alpha.

This differentiable process allows gradients to flow from the 2D image loss back to the 3D scene parameters.

A naive uniform sampling along each ray is computationally wasteful. NeRF employs a two-stage hierarchical sampling procedure to allocate samples efficiently:

1. Coarse Pass: A network (or the first half of a shared network) is evaluated at N_c uniformly sampled points. This produces a coarse density distribution.
2. Importance Sampling: The coarse densities are used to define a piecewise-constant PDF. A second set of N_f points is then sampled from this distribution, biasing samples towards regions likely containing visible surfaces (high density).

• This strategy concentrates computation where it matters most, dramatically improving rendering quality for a fixed total sample budget (N_c + N_f).
• It is a key innovation that separates NeRF from prior neural rendering methods.

The entire ray marching pipeline is fully differentiable, which is essential for training. The gradient of the photometric loss (e.g., MSE between rendered and ground truth pixel color) propagates backwards through:

• The alpha compositing (blending) equation.
• The neural network's density (σ) and color (c) predictions at each sample point.
• The positional encoding of the sampled 3D coordinates and viewing directions.

This end-to-end gradient flow enables the NeRF MLP to be optimized via gradient descent, simultaneously learning accurate geometry (via density σ) and appearance (via color c) from only 2D images and their camera poses.

Ray marching is the primary computational bottleneck in NeRF. Key parameters that govern the trade-off between speed and fidelity include:

• Number of Samples (N_c, N_f): More samples yield smoother, more accurate renders but increase query cost linearly. Typical values are 64 coarse and 128 fine samples.
• Ray Bounds (t_n, t_f): Defining tight near/far planes reduces wasted computation in empty space.
• Sample Stratification: Jittering sample positions within bins prevents aliasing but can introduce noise.

Acceleration techniques like InstantNGP's multi-resolution hash grid or TensorRF's factorization work by reducing the cost of each network query, allowing high-quality results with far fewer marched samples.

Standard ray marching treats rays as infinitesimally thin lines, causing aliasing when rendering at different resolutions or camera distances. Mip-NeRF addresses this by marching conical frustums instead of rays.

• For each pixel, it considers the volume of space its cone covers.
• It uses Integrated Positional Encoding (IPE) to encode the mean and variance of coordinates within each conical segment.
• The network then predicts the average color and density over that region.

This fundamental change to the ray marching primitive enables anti-aliasing, allowing a single NeRF model to render sharp details at any scale without artifacts like blurring or jagged edges.

Ray marching differs from other 3D rendering paradigms used in computer graphics and vision:

• vs. Rasterization (Triangle Meshes): Rasterization projects explicit geometry. Ray marching integrates an implicit, volumetric field. NeRFs have no polygons.
• vs. Ray Tracing: Ray tracing finds exact intersections with surfaces. Ray marching approximates intersections by stepping through a volume. It is necessary for continuous fields where an analytic intersection test doesn't exist.
• vs. Sphere Tracing (for SDFs): Sphere tracing uses the SDF value to make optimally large steps. Standard NeRF ray marching uses fixed or importance-sampled steps, as the density field isn't a signed distance function.

Ray marching is the natural and necessary choice for rendering the continuous 5D neural radiance field represented by a NeRF.

The overarching graphics technique that ray marching implements. Volume rendering generates a 2D image by integrating optical properties (color, density) along rays cast through a 3D volumetric field. The core equation is the volume rendering integral, which ray marching approximates numerically. This is the fundamental rendering equation used in NeRF and medical visualization.

• Key Input: A 3D scalar/vector field (e.g., density, radiance).
• Core Process: Accumulating light contributions along viewing rays.
• Output: A 2D pixel color.

A simpler precursor to ray marching used for surface rendering. Ray casting shoots a ray from the camera and finds the nearest surface intersection with explicit geometry (like a triangle mesh) using analytic solvers. It stops at the first hit. Ray marching, in contrast, is designed for volumetric fields where there is no explicit surface, requiring incremental sampling along the entire ray.

• Primary Use: Rendering polygonal meshes and heightfields.
• Difference: Uses geometric intersection tests; does not sample a continuous volume.

A specific type of implicit surface representation central to many ray marching implementations. An SDF is a continuous function that, for any point in space, returns the shortest distance to a surface, with the sign indicating inside (negative) or outside (positive). Ray marching with SDFs (sphere tracing) uses this distance as a safe step size, guaranteeing the ray never overshoots the surface, leading to highly efficient rendering.

• Core Property: f(x,y,z) = signed distance to surface.
• Application: Foundation of distance field rendering and real-time global illumination techniques.

An optimized, convergent form of ray marching designed specifically for rendering scenes defined by Signed Distance Functions (SDFs). At each step, the algorithm queries the SDF for the distance to the nearest surface and advances the ray by that exact distance. This guarantees the ray never penetrates the surface, allowing for far fewer steps than fixed-step ray marching.

• Key Advantage: Step size is dynamically maximized based on local geometry.
• Efficiency: Enables complex real-time scenes with <100 steps per ray.

The mathematical framework that makes ray marching essential for modern neural graphics like NeRF. Differentiable rendering allows gradients to flow from a synthesized 2D image back to the underlying 3D scene parameters (density, color, camera pose). The discrete sampling in ray marching is designed to be differentiable, enabling gradient-based optimization via backpropagation.

• Critical Link: Connects the rendering algorithm (ray marching) to the learning loop.
• Result: Allows a neural network to learn a 3D scene from 2D images by minimizing a photometric loss.

An adaptive sampling strategy used in advanced ray marching implementations, notably in NeRF. Instead of sampling uniformly, a two-pass process is used:

1. A coarse pass samples the ray at fixed intervals to estimate a density distribution.
2. A fine pass performs importance sampling, concentrating more samples in regions likely to contain visible content (high density).

• Purpose: Dramatically improves quality and efficiency by allocating computation where it matters most.
• Outcome: Reduces the total number of network queries needed for high-fidelity rendering.