Light Field

The most common representation models a light field as a 4D function, L(u, v, s, t), describing the radiance of every light ray. The ray is parameterized by its intersection with two parallel planes: an UV plane (aperture) and an ST plane (sensor). This two-plane parameterization provides a structured, non-redundant coordinate system for rays, enabling efficient storage and algorithms for refocusing and novel view synthesis.

This is a fundamental constraint in light field acquisition. For a fixed sensor resolution (e.g., a 10-megapixel chip), the captured data must be divided between spatial resolution (detail within a single view) and angular resolution (number of distinct viewpoints).

• High Angular, Low Spatial: Many slightly different views, each with lower detail. Ideal for smooth view interpolation.
• Low Angular, High Spatial: Fewer views, but each is a high-resolution image. This tradeoff is dictated by the plenoptic sampling theorem, which sets minimum sampling rates to avoid aliasing.

A fully sampled light field contains horizontal and vertical parallax, meaning the viewpoint can be shifted left-right and up-down. This enables the generation of novel views through view interpolation, where intermediate images are synthesized by selecting and blending appropriate rays from the captured set. The quality of this interpolation depends directly on the density of angular sampling. A critical challenge is occlusion handling, correctly managing scene regions hidden in some viewpoints but visible in others.

A direct application of the 4D light field is digital refocusing. After capture, the focal plane can be synthetically shifted by integrating subsets of rays that would have converged at a new virtual sensor plane. This process simulates changing the camera's depth of field without any physical lens movement. It relies on the shearing of the 4D light field data in ray space. This capability is a hallmark feature of consumer light field cameras like the Lytro.

Depth information is implicitly encoded in the light field's structure. In an epipolar plane image (EPI)—a 2D slice of the 4D field—a scene point appears as a line whose slope is inversely proportional to its depth. Disparity estimation across the angular dimensions provides a dense depth map without traditional stereo matching. This photo-consistency across multiple views is a powerful cue for 3D scene reconstruction and is foundational to multiview stereo algorithms.

Light fields are captured or displayed using specialized hardware:

• Plenoptic Cameras: Use a microlens array placed just before the sensor to sample angular information.
• Camera Arrays: A grid of synchronized cameras, providing high resolution but complex calibration.
• Integral Imaging Displays: Use a microlens array to reconstruct a light field for autostereoscopic 3D viewing, requiring no glasses.
• Holographic Stereograms: Synthesize a light field into a hologram, creating a full-parallax 3D image viewable under white light.

Digital refocusing allows the focal plane of a photograph to be adjusted after capture. By capturing the full 4D light field, a synthetic aperture can be applied computationally. This is achieved by integrating rays that converge at a chosen depth, effectively simulating a different lens focus setting. This capability is foundational to plenoptic cameras like the Lytro and is used in post-production and computational imaging pipelines.

Light fields enable the generation of novel viewpoints not present in the original capture. By re-sampling the ray space, images for arbitrary camera positions can be rendered. This is critical for:

• Autostereoscopic displays (glasses-free 3D) like those using integral imaging.
• Virtual reality and augmented reality content creation.
• Image-based rendering systems that generate fly-throughs from static captures. The quality depends on the density of angular sampling and effective occlusion handling.

The parallax encoded in a light field provides a robust signal for depth estimation. Techniques analyze the epipolar plane images (EPIs), where points at different depths manifest as lines with different slopes. This allows for dense depth maps to be extracted without explicit stereo matching. This application feeds directly into 3D scene reconstruction pipelines and is a core component of multiview stereo algorithms that leverage light field data.

Light fields facilitate HDR imaging by enabling the synthesis of images with varying virtual exposure settings from a single capture. Since the light field records radiance (not just pixel intensity), different subsets of rays can be combined to simulate longer or shorter exposures. This allows for the recovery of detail in both shadows and highlights without the need for multiple physical exposures or bracketing, mitigating motion ghosting artifacts.

By separating direct and indirect illumination components through analysis of the light field's directional data, it becomes possible to perform advanced scene editing. Applications include:

• Relighting: Modifying the direction and intensity of scene lighting.
• Material editing: Changing surface properties like glossiness or albedo.
• Glare and reflection removal: Using angular information to isolate and suppress specular highlights or reflections based on their directional consistency.

In scientific imaging, light field principles enable volumetric imaging from a single snapshot. In light field microscopy, a microlens array placed at the native image plane of a microscope allows the capture of 3D information about a sample. Computational processing then reconstructs a focal stack or full 3D volume. This enables high-speed 3D imaging of dynamic biological processes without mechanical scanning, trading some spatial resolution for volumetric capture speed.

The plenoptic function is the complete theoretical description of all visual information in a scene. It is a 7D function: V(x, y, z, θ, φ, λ, t), representing light intensity at every 3D point (x,y,z), for every direction (θ,φ), for every wavelength (λ), and at every time (t). A light field is a practical 4D or 5D slice of this function, typically assuming monochromatic or RGB light and a static scene.

• Foundation: Serves as the mathematical basis for all image-based rendering and light field theory.
• Dimensionality: The full 7D function is intractable to capture; light fields reduce dimensionality by making assumptions (e.g., outside convex hull, static scene).

A light field camera (or plenoptic camera) is a specialized imaging device that captures both the intensity and direction of light rays. It uses a microlens array placed between the main lens and the sensor. Each microlens creates a micro-image on the sensor, sampling the pupil plane of the main lens.

• Types: Plenoptic 1.0 (focused on microlenses) trades spatial resolution for angular data. Plenoptic 2.0 (focused on image plane) enables higher spatial resolution.
• Output: Captures a single raw image that is later processed to extract a 4D light field, generate sub-aperture images, or enable digital refocusing.

View synthesis is the core computational task of generating photorealistic images of a scene from arbitrary, novel camera viewpoints not present in the original capture. Light fields provide a direct, geometry-free path to view synthesis by re-sampling the captured ray database.

• Methods: Includes light field rendering (direct ray interpolation), image-based rendering (using proxy geometry), and modern neural rendering techniques like NeRF.
• Challenge: Requires dense angular sampling to avoid holes and maintain viewpoint consistency. Occlusion handling is a primary difficulty.

An Epipolar Plane Image is a powerful 2D visualization and analysis tool derived from a 4D light field. It is created by fixing one spatial dimension and one angular dimension, resulting in a 2D slice (u,x) or (v,y). In this slice, a scene point appears as a line whose slope is inversely proportional to its depth.

• Analysis: EPI analysis allows for depth estimation and occlusion reasoning directly from the light field data without explicit feature matching.
• Structure: The linear structures in an EPI directly encode the parallax between different views.

The spatial-angular tradeoff is a fundamental, unavoidable constraint in light field acquisition with a fixed sensor resolution. It states that for an N-megapixel sensor, the product of spatial resolution (x,y) and angular resolution (u,v) is roughly constant.

• Consequence: Increasing the number of sampled viewpoints (angular resolution) reduces the resolution of each individual view (spatial resolution).
• Plenoptic Sampling Theorem: Dictates the minimum sampling rates in both domains to avoid aliasing, formalizing this tradeoff. Super-resolution techniques are often needed to mitigate its effects.

The Lumigraph is a specific, structured 4D representation of a light field designed for efficient rendering. It parameterizes rays by their intersection with a bounding geometry (often a cube) surrounding the scene. This provides a scene-dependent parameterization versus the simple two-plane parameterization of a pure light field.

• Rendering: Enables faster novel view generation by using the coarse geometry to guide ray interpolation, improving performance on occlusion edges.
• Relation: Conceptually similar to a light field, but the Lumigraph explicitly incorporates approximate geometry to aid reconstruction, bridging image-based and geometry-based rendering.