Neural Radiance Field (NeRF)

Unlike traditional voxel grids or meshes, NeRF represents a scene as a continuous 5D function. A Multi-Layer Perceptron (MLP) maps spatial coordinates (x, y, z) and viewing direction (θ, φ) to a color and density value.

• Mechanism: The network learns to predict the radiance emitted at any point in space, allowing for the rendering of infinitely high-resolution images.
• Medical Relevance: This allows for the interpolation of anatomical structures between discrete CT or MRI slices, generating smooth, artifact-free 3D reconstructions without the stair-step artifacts common in standard interpolation.

NeRF's core capability is generating photorealistic 2D images from camera angles not present in the training set. It achieves this through classical volume rendering techniques combined with the learned neural representation.

• Process: Rays are cast through the scene, and the color and density at sampled points along each ray are accumulated to compute the final pixel color.
• Clinical Application: A NeRF trained on a few 2D X-ray projections can synthesize a full 3D CT-like volume, potentially enabling low-dose 3D imaging from standard biplanar radiography systems.

By conditioning the network's color output on the viewing direction, NeRF can model specularities and view-dependent appearances. In a medical context, this is adapted to model how different tissues interact with imaging modalities.

• Example: In endoscopic data, a NeRF can accurately re-render the glossy, wet surfaces of internal organs from any angle, improving the realism of virtual colonoscopies.
• Advanced Use: This property can be leveraged to model anisotropic tissue structures, such as muscle fibers or white matter tracts, where appearance changes based on the angle of the imaging plane.

The density field learned by a NeRF implicitly defines the 3D geometry of the scene. A surface mesh can be extracted by applying the marching cubes algorithm to the density grid.

• Surgical Planning: This allows for the automatic generation of highly accurate, watertight 3D models of patient-specific anatomy (e.g., a fractured bone or a tumor) directly from a set of images.
• Advantage: The resulting mesh is free from the topological errors and noise often found in direct segmentation-based reconstructions, providing a cleaner input for 3D printing and computational simulations.

Advanced NeRF variants can decompose a scene into separate components, such as bone and soft tissue, or even model the physical acquisition process itself.

• Bone-Soft Tissue Separation: A NeRF can be trained to output separate density fields for different tissue classes, enabling the digital subtraction of bone to visualize underlying vasculature without a second scan.
• Compensation for Artifacts: By modeling the imaging physics, a NeRF can be trained to correct for metal artifacts in CT scans, simultaneously reconstructing the anatomy and the distortion field caused by implants.

Standard NeRF assumes a static scene. Deformable NeRF architectures introduce a deformation field that maps points from a canonical 3D space to the observed scene at each time instant.

• Cardiac Imaging: This is critical for modeling the beating heart, where a NeRF can learn a canonical, static representation of the heart and a time-varying deformation field to capture its motion.
• Result: This enables the reconstruction of a high-quality 4D (3D + time) cardiac model from a single, undersampled cine MRI acquisition, dramatically reducing scan time while improving temporal resolution.