Novel View Synthesis

Neural Radiance Fields (NeRF) is a foundational deep learning technique that represents a scene as a continuous volumetric function. It uses a multilayer perceptron (MLP) to map a 3D spatial coordinate and 2D viewing direction to a volume density and view-dependent RGB color. The scene is rendered via differentiable volume rendering, and the network is optimized by minimizing the photometric error between synthesized and ground-truth input views.

• Core Innovation: Replaces discrete 3D representations (meshes, point clouds) with a continuous, memory-efficient neural field.
• Key Limitation: Slow training and rendering due to the need to query the MLP millions of times per image.
• Example: Synthesizing a 360-degree walkthrough of a room from a few dozen photos.

3D Gaussian Splatting is a state-of-the-art, real-time rendering technique that represents a scene as a collection of anisotropic 3D Gaussians. Each Gaussian has attributes for position, covariance (scale/rotation), opacity, and spherical harmonics for view-dependent color. For rendering, these 3D Gaussians are projected to 2D and splatted onto the image plane using a tile-based rasterizer.

• Core Advantage: Achieves photorealistic quality at real-time frame rates (> 100 FPS), enabling interactive applications.
• Workflow: Starts with a point cloud from Structure from Motion (SfM), initializes a Gaussian per point, and optimizes them via differentiable rendering.
• Use Case: Real-time visualization for virtual reality, architectural walkthroughs, and digital twins.

Image-Based Rendering (IBR) is a classical family of techniques that synthesize novel views by warping and blending pixels from nearby input images, without explicitly reconstructing a detailed 3D model. It relies on geometric proxies—such as rough depth maps or planar surfaces—to guide the pixel reprojection.

• Key Methods: Include Light Field Rendering (capturing dense image arrays) and Depth-Image-Based Rendering (DIBR).
• Advantage: Can produce highly realistic results when input views are dense, as it directly uses captured light.
• Limitation: Quality degrades significantly with sparse inputs or inaccurate geometric proxies.
• Historical Context: The precursor to modern neural methods, used in early panorama stitching and view interpolation.

This traditional pipeline first uses Multi-View Stereo (MVS) to generate an explicit 3D reconstruction (a dense point cloud or mesh) from calibrated images. A novel view is then synthesized by projecting this 3D geometry into the new camera and texturing it by blending colors from visible input images.

• Explicit Geometry: Produces an interpretable, editable 3D asset (e.g., a .ply mesh).
• Challenges: Struggles with non-Lambertian surfaces (reflections, transparency) and requires solving complex visibility and blending problems.
• Software: Implemented in tools like COLMAP, OpenMVS, and Meshroom. Often used as a preprocessing step for neural methods to obtain camera poses and initial geometry.

This approach represents scene geometry using a neural implicit representation, most commonly a Signed Distance Function (SDF). A neural network maps 3D coordinates to the signed distance to the nearest surface. This continuous representation is often paired with a separate network for color and rendered using differentiable ray marching.

• Advantage over NeRF: Produces cleaner, watertight surface geometry which is crucial for robotics and CAD applications.
• Methods: Includes NeuS, VolSDF, and Instant-NGP with an SDF base.
• Output: The zero-level set of the learned SDF can be extracted as a high-quality mesh via Marching Cubes.

This class of techniques uses generative models, like diffusion models or Generative Adversarial Networks (GANs), to synthesize novel views, often from a single input image. Instead of reconstructing a precise 3D model, they learn the manifold of plausible appearances from large datasets.

• Core Capability: Can hallucinate geometrically and texturally consistent views for objects or scenes not fully observed.
• Input Flexibility: Often designed for sparse or single-image input, making them more applicable to casual photography.
• Trade-off: May sacrifice strict geometric accuracy for perceptual plausibility and generalization.
• Example Models: Zero-1-to-3, SyncDreamer, and various 3D-aware GANs like EG3D.

A foundational deep learning technique for novel view synthesis. NeRF represents a scene as a continuous volumetric scene function—a neural network that maps a 3D location and viewing direction to color and volume density. By optimizing this network from a set of posed 2D images, it can render highly photorealistic novel views, including complex effects like view-dependent specularities and semi-transparency. It is the basis for many subsequent real-time and efficient variants.

A state-of-the-art, real-time rendering technique for novel view synthesis. It represents a scene as a collection of millions of anisotropic 3D Gaussians, each with attributes for position, covariance (scale/rotation), opacity, and spherical harmonics for view-dependent color. During rendering, these Gaussians are projected to 2D and splat onto the image plane using a differentiable tile-based rasterizer. This approach achieves quality rivaling NeRF but at interactive frame rates, making it suitable for dynamic applications.

A classical photogrammetry pipeline that provides the camera poses and sparse 3D point cloud essential for many novel view synthesis methods. SfM works by:

• Detecting distinctive keypoints (e.g., SIFT, ORB) in a collection of unordered images.
• Matching these keypoints across images.
• Solving a geometric optimization problem to estimate camera parameters and 3D point positions. The output is a calibrated set of images, which serves as the primary input for techniques like NeRF and Multi-View Stereo.

The process of generating a dense 3D reconstruction from multiple calibrated images. While SfM produces a sparse point cloud, MVS aims to estimate depth for every (or most) pixels, resulting in a dense point cloud or mesh. Key approaches include:

• Patch-based methods that match small image patches.
• Depth-map fusion that estimates a depth map per view and merges them.
• Volumetric methods that use a cost volume in 3D space. The dense geometry from MVS can be used directly or refined by neural rendering techniques for higher-quality synthesis.

A computational framework that allows gradients to be propagated from a rendered 2D image back to the underlying 3D scene parameters (e.g., geometry, materials, lighting). This is the enabling engine behind learning-based 3D reconstruction and novel view synthesis. By making the rendering process differentiable, optimization techniques can adjust 3D representations to minimize a loss between synthesized images and ground-truth photographs. It is the core mechanism that allows NeRF and similar models to be trained from images alone.

A paradigm for representing 3D shapes and scenes using a neural network (typically a multilayer perceptron) as a continuous function. Instead of storing explicit data structures like meshes or voxel grids, the network encodes the scene by mapping 3D coordinates to properties such as:

• Signed Distance (SDF): Distance to the nearest surface.
• Occupancy: Probability a point is inside matter.
• Radiance Fields (as in NeRF): Color and density. This representation is compact, infinitely resolvable, and naturally suited for gradient-based optimization via differentiable rendering.