Proposal Network

The seminal NeRF paper introduced a two-stage proposal network architecture. A coarse network first samples 64 points along each ray to produce a preliminary density distribution. This distribution then guides a fine network, which performs importance sampling of 128 additional points concentrated in high-density regions. This hierarchical approach reduces the total number of samples required for high-quality rendering from thousands to under 200 per ray.

For unbounded, 360-degree scenes, Mip-NeRF 360 employs a chain of lightweight proposal samplers. Key innovations include:

• Proposal loss: A loss function that ensures each proposal sampler's histogram matches the final rendered distribution, preventing collapse.
• Distillation: Each sampler is trained to predict the output of the next, more refined sampler.
• This multi-stage cascade allows the final NeRF model to sample sparsely (typically 32 points) in empty space and densely in occupied regions, enabling real-time performance for complex environments.

Instant Neural Graphics Primitives (Instant NGP) replaces a neural proposal network with an explicit, trainable density grid. This grid acts as a highly efficient proposal mechanism:

• The grid stores a density value at each voxel, updated via exponential moving average during training.
• During ray marching, the grid is queried to skip empty space via early termination and to sample more densely in occupied voxels.
• This explicit structure provides faster lookups than a small MLP, contributing significantly to Instant NGP's millisecond-scale training and rendering times.

While not a network in the traditional sense, 3D Gaussian Splatting employs a spatial data structure as a proposal mechanism. An initial point cloud from Structure-from-Motion (SfM) provides a strong prior. During optimization and rendering:

• A bounding volume hierarchy (BVH) is used to efficiently cull Gaussians not intersecting a pixel's frustum.
• The adaptive density control algorithm, which clones and prunes Gaussians, effectively 'proposes' where more or less detail is needed in the 3D scene representation.

In a deferred neural rendering pipeline, the proposal network's role is often fulfilled by the geometry pass. The rasterized G-buffer (containing depth, normals, and material IDs) defines the surfaces to be shaded. A subsequent neural network (e.g., a neural texture decoder) uses these surfaces as its 'proposal' for where to apply complex, view-dependent shading, bypassing volumetric ray marching entirely for opaque geometry.

For dynamic neural radiance fields, proposal networks must account for time. Systems like HyperNeRF or D-NeRF often use separate proposal mechanisms for canonical space and deformation. A proposal network may first sample in a learned canonical 3D volume. A deformation field network then maps these samples to their time-dependent positions before evaluation by the main radiance field, ensuring sampling efficiency is maintained across temporal sequences.

The foundational volumetric rendering algorithm where a view ray is incrementally stepped through a 3D scene. At each sample point, a density and color are queried. The proposal network's primary role is to predict an importance sampling distribution that tells the ray marcher where to sample most effectively, skipping empty space and concentrating on opaque surfaces.

A Monte Carlo integration technique critical for efficient neural rendering. Instead of sampling a ray uniformly, samples are concentrated in regions that contribute most to the final pixel color (e.g., high-density areas). The proposal network learns to predict this optimal distribution, which:

• Reduces variance and noise in the rendered image.
• Cuts required samples by 10x or more, enabling real-time frame rates.
• Works in a coarse-to-fine manner with the main rendering network.

An explicit-neural hybrid data structure used to encode spatial features. It is the typical feature encoder for a proposal network in frameworks like Instant NGP. It consists of multiple hash tables at different resolutions, providing:

• Fast, O(1) lookup of features for any 3D coordinate.
• Compact memory usage due to hash collision handling.
• Multi-scale feature capture, allowing the network to learn both fine details and coarse geometry for sampling.

The overarching architectural pattern for real-time NeRF systems. It combines an explicit data structure (like a hash grid or sparse voxel grid) with a small neural network. The proposal network is a prime example:

• Explicit Component: The multi-resolution hash grid stores learnable spatial features.
• Neural Component: A tiny MLP (e.g., 1-2 layers) decodes these features into a sampling probability distribution.
• This hybrid approach balances the speed of grid lookups with the representational flexibility of neural networks.

The hierarchical, two-pass strategy enabled by the proposal network. It is the core of the Mip-NeRF 360 and Instant NGP rendering pipelines:

1. Coarse Pass: The proposal network evaluates the ray at a few samples to predict a piecewise-constant PDF (Probability Density Function).
2. Fine Pass: The main, high-capacity rendering network uses inverse transform sampling on this PDF to draw its samples, focusing computational effort on relevant regions. This decoupling is key to rendering efficiency.

An alternative real-time rendering paradigm where a proposal network is not used. Instead, a traditional rasterizer first produces a G-buffer (containing depth, normals, etc.). A neural network then processes this 2D buffer to produce the final image, applying effects like neural textures or global illumination. This contrasts with the proposal network's volumetric, ray-based approach, trading full 3D reasoning for guaranteed frame rates.