Neural Radiance Field (NeRF)

Unlike explicit 3D representations such as meshes, point clouds, or voxel grids, NeRF encodes a scene within the weights of a fully-connected deep network. The model maps a continuous 5D coordinate—spatial location (x, y, z) and viewing direction (θ, φ)—to an emitted color and volume density at that point. This continuous formulation allows the scene to be queried at arbitrary resolutions without being constrained by discrete grid boundaries, making it exceptionally memory-efficient for complex geometries.

To synthesize an image from a novel viewpoint, NeRF casts a camera ray through each pixel and samples points along its path. The classical volume rendering equation accumulates the radiance contributions of these samples, weighted by their transmittance. Because the entire rendering process is formulated as a differentiable function, gradients can flow from the pixel-level photometric loss back through the ray marcher to update the network weights. This end-to-end differentiability is the core mechanism that enables learning from only 2D supervision.

Standard deep networks exhibit a spectral bias toward learning low-frequency functions, resulting in blurry reconstructions. NeRF addresses this by mapping input coordinates to a higher-dimensional space using a sinusoidal positional encoding (Fourier feature mapping). This transformation dramatically improves the model's ability to represent sharp edges, fine textures, and high-frequency geometric detail. The encoding function is defined as:

• γ(p) = [sin(2⁰πp), cos(2⁰πp), ..., sin(2^{L-1}πp), cos(2^{L-1}πp)]
• The hyperparameter L controls the maximum frequency captured.

NeRF decouples geometry from appearance by conditioning the emitted color prediction on the viewing direction while predicting volume density solely from spatial location. This architectural inductive bias enforces multi-view consistency: the shape of an object (density) remains invariant regardless of the observer's angle, but its appearance (color) can vary to capture specular highlights, reflections, and other non-Lambertian effects. This separation is critical for photorealistic rendering of shiny or translucent materials.

Evaluating the network at every point along a ray is computationally prohibitive. NeRF employs a two-stage coarse-to-fine sampling strategy:

• A coarse network evaluates a uniform set of sample points to produce a rough probability density function along the ray.
• A fine network then allocates samples proportionally to this density, concentrating computation on surfaces and occupied regions rather than empty space. This hierarchical approach focuses model capacity where it matters most, significantly improving both training efficiency and rendering quality.

In medical imaging, NeRF-based methods are being adapted to reconstruct continuous 3D anatomical volumes from sparse 2D X-ray projections or limited-angle CT acquisitions. Unlike traditional Filtered Back Projection or Iterative Reconstruction, implicit neural representations can learn strong anatomical priors from data, enabling high-quality reconstructions from ultra-sparse views. This has profound implications for reducing radiation dose in intraoperative imaging and enabling portable, low-cost CT systems. Frameworks like MedNeRF and NeAT extend the core algorithm to handle non-linear radiographic formation models.

A volumetric pixel representing a value on a regular grid in 3D space. It is the fundamental unit for CT and MRI image reconstruction, analogous to a 2D pixel but with depth.

• Represents a scalar intensity (e.g., Hounsfield Unit).
• Forms the discrete grid for volumetric data.
• NeRFs learn a continuous function, effectively predicting radiance and density at any 3D coordinate, moving beyond discrete voxel grids.

A visualization technique that projects a 3D volumetric dataset directly onto a 2D viewing plane. It assigns color and opacity to each voxel via a transfer function.

• Ray casting: The core algorithm for sampling the volume along viewing rays.
• NeRFs use a differentiable volume rendering equation to composite color and density samples along a ray, enabling training from 2D images.

A classic computer graphics algorithm for extracting a polygonal mesh of an isosurface from a discrete scalar field. It processes the volume voxel-by-voxel to generate triangles.

• Widely used for surface rendering of anatomical structures from segmentation masks.
• After training a NeRF, Marching Cubes can be applied to the learned density field to extract an explicit 3D mesh for further analysis or simulation.

A non-linear spatial alignment technique that warps a moving image to match a fixed image. It accounts for anatomical variations, tissue compression, or physiological motion.

• Uses a dense deformation vector field to map coordinates.
• Deformable NeRFs extend the standard model to handle dynamic scenes, making them suitable for reconstructing moving anatomy like a beating heart from sparse views.

A class of CT and MRI reconstruction algorithms using deep neural networks to suppress noise and resolve fine structures beyond conventional limits.

• Trained on paired low-quality and high-quality data.
• NeRF represents a novel form of DLR, using an implicit neural representation to solve the inverse problem of reconstructing a 3D volume from its 2D projections, often with superior novel view synthesis.