Neural Radiance Fields (NeRF)

A NeRF encodes a scene as a continuous 5D function that outputs a volume density (σ) and view-dependent RGB color (c) for any 3D spatial coordinate (x, y, z) and 2D viewing direction (θ, φ). This is mathematically expressed as: F_Θ: (x, y, z, θ, φ) → (c, σ). The neural network, typically a Multilayer Perceptron (MLP), acts as a compact, differentiable database that can be queried at any point in space, enabling the generation of infinitely sharp details without the discretization artifacts of voxel grids or point clouds.

To synthesize a 2D image, NeRF uses classical volume rendering principles made differentiable. For each pixel, a ray is cast into the scene, and points are sampled along it. The final pixel color Ĉ(r) is computed by integrating the colors and densities of these samples:

Ĉ(r) = Σᵢ Tᵢ (1 - exp(-σᵢ δᵢ)) cᵢ

where:

• Tᵢ = exp(-Σⱼ₌₁ⁱ⁻¹ σⱼ δⱼ) is the transmittance (how much light reaches sample i).
• σᵢ, cᵢ are the network's predictions.
• δᵢ is the distance between samples.

This differentiable rendering equation allows gradients from a 2D photometric loss (e.g., Mean Squared Error against input images) to flow backwards through the integration process to optimize the underlying 5D neural scene representation.

A standard MLP struggles to represent high-frequency details in scenes. NeRF overcomes this by applying a fixed, high-dimensional positional encoding γ(p) to the input 3D coordinates and viewing directions before passing them to the network. For a value p (normalized to [-1, 1]), the encoding is:

γ(p) = (sin(2⁰πp), cos(2⁰πp), ..., sin(2ᴸ⁻¹πp), cos(2ᴸ⁻¹πp))

• For spatial coordinates (x,y,z), L=10 is typical.
• For viewing direction, L=4 is typical. This mapping projects low-dimensional inputs into a higher-dimensional space, enabling the MLP to more easily approximate functions with fine spatial and angular variation, which is critical for modeling sharp edges, textures, and specular highlights.

Efficiently sampling along a ray is critical. Naïve uniform sampling is wasteful, as most of empty space contributes nothing. NeRF uses a two-stage, hierarchical sampling strategy:

1. Coarse Network: A first MLP is queried with a set of uniformly distributed samples along the ray to produce an initial density estimate.
2. Fine Network: These densities are used to produce a piecewise-constant PDF, which informs a second, more focused set of samples. This importance sampling concentrates evaluations in regions likely to contain visible surfaces.

Both coarse and fine outputs are rendered, and the combined loss ensures the coarse network learns a good prior for empty/occupied space, dramatically improving sample efficiency and final render quality.

A key innovation is the separation of diffuse geometry (density σ) from view-dependent color (RGB c). The volume density σ is predicted solely from the spatial coordinate (x,y,z), making it independent of viewpoint. The color c, however, is predicted from both the spatial coordinate and the viewing direction (θ, φ). This allows the network to model non-Lambertian effects like specular highlights, reflections, and transparency, which change appearance based on the camera angle. This is why NeRF can accurately reproduce materials like glass, metal, or wet surfaces from multi-view imagery.

Unlike explicit representations (meshes, point clouds, voxel grids), a NeRF is an implicit neural representation. The scene is not stored as discrete elements but as the weights of the MLP. This offers significant advantages:

• Memory Efficiency: A scene can be represented by a few megabytes of network weights, rather than gigabytes of dense point clouds or high-resolution voxel grids.
• Infinite Resolution: The continuous function can be queried at arbitrary spatial precision, avoiding the fixed resolution limits of voxels.
• Natural Smoothness & Interpolation: The MLP provides a smooth prior, filling in plausible geometry and appearance in regions with sparse observations, acting as a powerful regularizer.
• Differentiability: The entire representation is end-to-end differentiable, enabling optimization from images alone.

A real-time rendering technique that represents a 3D scene as a collection of anisotropic 3D Gaussians. These primitives are optimized from images and can be differentiably rendered via splatting to produce photorealistic novel views at high frame rates. It is considered a key successor to NeRF, trading some visual fidelity for orders-of-magnitude faster training and rendering.

• Key Differentiator: Explicit, discrete representation vs. NeRF's implicit, continuous function.
• Primary Use Case: Real-time applications like VR/AR and interactive scene viewing.

A continuous volumetric representation where the value at any 3D point is its signed distance to the nearest object surface. Negative values indicate inside an object, positive values indicate outside. Neural implicit representations often use an MLP to encode an SDF, which can then be converted into a mesh via Marching Cubes. Unlike NeRF's density/color output, an SDF provides a clean, watertight geometric surface.

• Relation to NeRF: Methods like NeuS and VolSDF combine SDF-based geometry with NeRF-style view-dependent appearance.
• Application: High-quality mesh extraction for CAD and simulation.

The core computer vision and graphics task of generating a photorealistic image of a scene from a previously unseen camera viewpoint. NeRF is a premier technique for this task, but it is one of many approaches. The field includes classical methods like Image-Based Rendering (IBR) and modern learning-based methods. The goal is to achieve photorealism, view consistency, and handling of complex lighting.

• Input: A set of posed images.
• Output: A new, plausible image from a novel camera pose.

A framework that allows gradients to be computed from a rendered 2D image back to the underlying 3D scene parameters (geometry, materials, lighting). This is the enabling technology behind NeRF's optimization. By making the rendering process differentiable, a loss between a synthesized image and a real image can be used to update the neural network representing the 3D scene.

• Core Mechanism: The rendering equation is made differentiable.
• Applications: Beyond NeRF, used for inverse graphics, material estimation, and mesh optimization.

A classical photogrammetry pipeline that estimates sparse 3D point clouds and camera poses from a collection of unordered 2D images. It works by detecting and matching keypoint features (e.g., SIFT) across images and solving a geometric optimization problem. SfM is often a preprocessing step for NeRF, providing the necessary camera calibration data.

• Output: Sparse point cloud and camera parameters.
• Key Algorithm: Bundle Adjustment for joint non-linear optimization.

A computer vision technique that takes the calibrated camera poses from SfM and generates a dense 3D reconstruction, such as a point cloud or mesh. It works by finding pixel correspondences across multiple views to triangulate depth. While NeRF produces a continuous radiance field, traditional MVS outputs explicit, often noisy, geometric data that requires post-processing.

• Relation to NeRF: Solves a similar problem (3D from images) but with different representations and outputs.
• Common Output: Dense point cloud or textured mesh.