View Synthesis

Image-Based Rendering generates novel views directly from a set of captured photographs, often with minimal or approximate 3D geometry. It warps and blends pixels from nearby input views to the target viewpoint.

• Core Principle: Relies on photo-consistency and geometric proxies (like depth maps) for pixel reprojection.
• Key Techniques: Include light field rendering (using a dense 4D ray dataset), Lumigraph rendering (structured around a coarse 3D hull), and view interpolation.
• Trade-off: Can produce highly photorealistic results from dense inputs but struggles with disocclusions (revealing unseen areas) if sampling is sparse.
• Example: The classic Stanford Light Field Archive demonstrates IBR by allowing interactive navigation from a dense grid of camera images.

This approach first reconstructs an explicit 3D model of the scene (mesh, point cloud, or voxel grid) from multi-view images, then uses traditional computer graphics rasterization or ray tracing to render novel views.

• Core Pipeline: Involves camera pose estimation, dense multiview stereo matching, surface reconstruction, and texture mapping.
• Strengths: Provides a compact, interpretable 3D asset that supports robust occlusion handling and lighting changes.
• Limitations: The reconstruction step can fail on textureless or reflective surfaces, and the final render quality is bounded by the fidelity of the recovered geometry and textures.
• Example: Photogrammetry software like RealityCapture or COLMAP epitomizes this geometry-first pipeline.

Neural Radiance Fields represent a scene as a continuous, implicit function parameterized by a multilayer perceptron (MLP). The network maps a 3D location and viewing direction to a volume density and view-dependent RGB color.

• Core Innovation: Uses differentiable volume rendering to optimize the MLP from a set of posed images without any explicit 3D supervision.
• Output: A neural scene representation that can be queried to synthesize novel views with fine detail and complex view-dependent effects like specular highlights.
• Challenge: Original formulations were computationally expensive to train and query, leading to extensive research in real-time variants.
• Example: The seminal paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (Mildenhall et al., 2020) introduced this paradigm.

Modern, high-performance methods combine explicit data structures with neural networks to accelerate NeRF. They trade pure implicit continuity for speed and scalability.

• Core Idea: Store features (e.g., colors, densities) in explicit, queryable structures like hash grids, sparse voxel octrees, or tensor decompositions. A small MLP decodes these features into final color/density.
• Benefits: Dramatically reduces the number of MLP evaluations, enabling real-time neural rendering (e.g., >30 FPS).
• Key Methods: Instant Neural Graphics Primitives (InstantNGP) uses a multi-resolution hash table. Plenoxels uses a sparse voxel grid without an MLP. 3D Gaussian Splatting uses anisotropic 3D Gaussians as explicit primitives.
• Use Case: Essential for interactive applications in AR/VR and spatial computing.

When input views are extremely sparse (e.g., a single image), the problem becomes severely under-constrained. Generative approaches use priors learned from large datasets to hallucinate plausible geometry and appearance.

• Core Technology: Leverages generative models like diffusion models or Generative Adversarial Networks (GANs) trained on massive 3D or multi-view datasets.
• Function: The model learns the manifold of realistic 3D objects/scenes, allowing it to infer missing information and ensure multi-view consistency from limited inputs.
• Applications: Single-image 3D reconstruction, content creation from text prompts, and inpainting disoccluded regions in view synthesis.
• Example: Zero-1-to-3 is a diffusion model that learns a prior for relative camera pose conditioning, enabling novel view synthesis from a single image.

A cross-cutting constraint, not a standalone method, that is fundamental to all high-quality view synthesis. It ensures the synthesized scene is coherent from all possible viewpoints.

• Definition: The property that the rendered appearance and geometry of a 3D point should be consistent across all input views where it is visible.
• Enforcement Mechanisms:
  • In classical MVS: Photo-consistency metrics (NCC, SSIM).
  • In NeRF: The volume rendering integral inherently optimizes for multi-view consistency via gradient descent.
  • In generative models: Often baked into the training objective via adversarial or diffusion losses across views.
• Failure Modes: Violations appear as "floaters," blurring, or flickering geometry that changes unrealistically with viewpoint. Robust occlusion handling is key to maintaining it.

The plenoptic function is the complete theoretical description of all light in a scene. It is a 7D function: P(x, y, z, θ, φ, λ, t) representing light intensity at every 3D position (x,y,z), for every direction (θ, φ), for every wavelength λ, and at every time t. It is the foundational model from which all view synthesis and light field techniques are derived.

• Complete Scene Representation: Captures the entirety of visual information.
• Dimensionality Reduction: Practical systems sample a lower-dimensional slice (e.g., 4D for a static light field).
• Theoretical Basis: Informs the sampling requirements and limits of image-based rendering systems.

A light field is a practical, sampled representation of the plenoptic function, typically parameterized as a 4D function L(u, v, s, t). It describes the radiance of light rays passing through two planes in space. This is the core data structure for many view synthesis methods.

• 4D Representation: The two-plane parameterization is standard for capture and processing.
• Ray-Based Model: Each sample corresponds to a single ray of light with a specific position and direction.
• Applications: Enables digital refocusing, viewpoint change, and depth estimation without explicit 3D geometry.

Image-Based Rendering is a class of computer graphics techniques that generate novel views directly from a set of input images, often bypassing the explicit reconstruction of a detailed 3D model. It is a broader category that encompasses many view synthesis approaches.

• Geometry vs. Image Trade-off: Methods range from pure interpolation (no geometry) to those using approximate depth (some geometry).
• Efficiency: Can produce photorealistic results more efficiently than traditional polygonal rendering for complex scenes.
• Key Techniques: Includes light field rendering, lumigraph rendering, and view morphing.

Novel View Synthesis is the specific task of generating a photorealistic image of a scene from a camera viewpoint not present in the original input set. It is the primary objective and evaluation benchmark for view synthesis systems.

• Core Challenge: Requires reasoning about occlusions, disocclusions, and non-Lambertian surfaces.
• Evaluation Metrics: Uses metrics like PSNR, SSIM, and LPIPS to compare synthesized views to ground-truth imagery.
• Modern Drivers: Heavily advanced by neural rendering techniques like Neural Radiance Fields (NeRF).

Multiview Stereo is a computer vision technique that estimates dense 3D geometry (a point cloud or mesh) from a collection of overlapping 2D images with known camera poses. It is a traditional, geometry-first precursor to many neural view synthesis methods.

• Dense Correspondence: Solves the correspondence problem across many images to triangulate 3D points.
• Photo-Consistency: Relies on the assumption that a correct 3D point will project to similarly colored pixels in all visible views.
• Pipeline Component: Often used to generate initial depth estimates or geometric proxies for more advanced IBR.

Epipolar geometry describes the intrinsic projective relationship between two views of a scene. It provides the geometric constraints that simplify the search for corresponding points between images, which is fundamental for stereo vision and many view synthesis algorithms.

• Epipolar Lines: A point in one image corresponds to a line in the other image, reducing the correspondence search from 2D to 1D.
• Essential & Fundamental Matrices: Mathematical models that encode this relationship.
• Critical for Consistency: Enforces multi-view consistency, a key constraint in optimizing neural scene representations.