Neural Radiance Fields (NeRF)

A NeRF models a scene as a continuous 5D function. For any 3D spatial coordinate (x, y, z) and 2D viewing direction (θ, φ), the model outputs:

• Volume density (σ): A scalar representing the differential probability of a ray terminating at that point (i.e., occupancy).
• RGB color (c): The view-dependent emitted radiance at that point.

This continuous representation, encoded by a multilayer perceptron (MLP), enables the synthesis of infinitely sharp details without the discretization artifacts of voxel grids or point clouds. It inherently represents empty space, surfaces, and semi-transparent volumes within a single unified model.

To generate a 2D image from the 5D NeRF, a differentiable volume rendering technique is used. For each pixel:

1. A camera ray is cast into the scene.
2. The ray is sampled at multiple 3D points along its length.
3. The NeRF MLP predicts density and color for each sample.
4. Colors are alpha-composited along the ray using the classic rendering equation, approximating the integral of accumulated transmittance.

The key innovation is that this entire pipeline—from 3D coordinates to final pixel color—is end-to-end differentiable. This allows the model to be trained from only 2D images by minimizing the photometric loss between rendered and ground-truth pixels, with gradients flowing back through the rendering process to optimize the MLP's weights.

Unlike simpler 3D representations, a NeRF captures non-Lambertian surface properties. By taking the viewing direction (θ, φ) as an additional input to the color head of the MLP, it can model:

• Specular highlights that change with camera angle.
• Reflections on glossy surfaces.
• Subtle translucency and subsurface scattering effects.

This allows synthesized novel views to exhibit realistic lighting variations, which is critical for photorealism. The network learns to disentangle geometry (density, which is view-independent) from complex, view-dependent appearance.

The geometry of the scene is implicitly defined by the learned volume density field σ(x, y, z). A surface is not explicitly stored as a mesh; instead, it exists where the density field exhibits a sharp transition from low to high values.

• Surface extraction: An explicit mesh (e.g., for use in CAD or game engines) can be extracted post-hoc by applying the Marching Cubes algorithm to a thresholded density field.
• Advantage: This avoids topology limitations of explicit representations, allowing modeling of complex, fuzzy, or fluid objects.
• Challenge: The implicit nature makes direct spatial queries or edits less straightforward than with a mesh or point cloud.

A standard NeRF is trained on a sparse set of multi-view images (typically dozens to hundreds) of a static scene. Critical prerequisites are:

• Known camera poses: The exact position, orientation, and intrinsic parameters (focal length) for each input image must be provided, often obtained via Structure-from-Motion (SfM).
• Consistent lighting: The scene illumination is assumed static across all input views.

The training process performs a form of multi-view stereo (MVS) internally, but at a per-scene optimization level. The model learns to reconcile photometric consistency across all input views to infer a coherent 3D structure, effectively solving the correspondence problem through gradient descent.

The canonical NeRF architecture has significant computational demands:

• Training: Requires hours to days on a single high-end GPU due to the need to query the MLP millions of times per ray for millions of rays.
• Inference/Rendering: Synthesizing a single novel view is slow (seconds to minutes) because it requires evaluating the MLP at thousands of samples for each of the image's hundreds of thousands of pixels.

This characteristic has driven a major research subfield focused on accelerating NeRF, leading to techniques like:

• InstantNGP: Using multi-resolution hash tables for fast feature lookup.
• 3D Gaussian Splatting: A fundamentally different, rasterization-based approach for real-time rendering.
• Plenoxels: A sparse voxel grid representation that bypasses the MLP for faster optimization.

A Signed Distance Field (SDF) is a volumetric representation where the value at any 3D coordinate represents the shortest distance to the surface of an object, with the sign indicating interior (negative) or exterior (positive). While a standard NeRF models volume density and color, SDF-based variants (like NeuS or VolSDF) use an SDF to define the geometry, which provides a cleaner, watertight surface representation ideal for mesh extraction. This is critical for robotics and CAD applications where an explicit, high-quality surface is required for collision detection or manufacturing. The zero-level set of the SDF (where the value is 0) defines the object's surface.

Multi-View Stereo (MVS) is a classical computer vision technique for dense 3D reconstruction from multiple overlapping 2D images. It operates by finding pixel correspondences across images and performing triangulation to estimate 3D points. Unlike NeRF, which learns a continuous radiance field, traditional MVS produces a discrete point cloud or depth maps. However, modern learning-based MVS methods (e.g., MVSNet) use cost volumes and CNNs to improve robustness. MVS is often used as a preprocessing step to provide geometric priors for NeRF training, significantly speeding up convergence and improving results in textureless or specular regions.

Neural surface representations are a family of techniques that explicitly model the surface of objects rather than the entire volume. This includes methods like:**

• NeuS: Uses a signed distance function (SDF) parameterized by a neural network.
• VolSDF: Incorporates an SDF into a volume rendering framework with a transformed density function. The key advantage over standard density-based NeRF is the ability to extract high-fidelity meshes directly via marching cubes on the learned SDF, without the need for thresholding a noisy density field. This makes them essential for applications in robotics (for collision meshes) and computer graphics (for asset creation).

Photogrammetry is the science of making measurements from photographs. Structure from Motion (SfM) is a key photogrammetry technique that solves for sparse 3D point clouds and camera poses from a collection of unordered 2D images. Tools like COLMAP are the de facto standard first step in a NeRF pipeline. SfM provides the necessary data for NeRF:

• Camera calibration (intrinsics like focal length).
• Camera poses (extrinsics: position and orientation for each image).
• Sparse 3D points for scene bounds. Without accurate SfM output, a NeRF model will fail to converge or produce a coherent 3D scene. It bridges raw images to the posed image datasets NeRF requires.