Ray Space

The Two-Plane Parameterization (2PP), formalized by Levoy and Hanrahan, is the most prevalent model for light fields. It represents each ray by its intersection coordinates (u, v) on a first plane and (s, t) on a second, parallel plane. This creates a natural 4D function L(u, v, s, t) where (u, v) defines a spatial location and (s, t) defines an angular direction.

• Core Concept: The ray passes through point (u, v) on the UV (camera) plane and point (s, t) on the ST (focal) plane.
• Rendering: Novel views are generated by extracting the appropriate 2D slice (s, t) for each pixel, effectively re-sorting captured rays.
• Applications: Forms the basis for the Lumigraph and is the standard model for most light field camera designs and research datasets.

The Sphere Parameterization describes a ray by its point of origin on a sphere and its direction vector. This is often represented as L(θ, φ, Θ, Φ), where (θ, φ) are spherical coordinates for the origin and (Θ, Φ) for the direction.

• Core Concept: Ideal for capturing full spherical light fields, such as those needed for immersive 360° environments or for scenes observed from the interior of a bounding sphere.
• Mathematical Utility: Aligns naturally with spherical harmonics and other frequency-domain analyses of the plenoptic function.
• Challenge: Dense, uniform sampling on a sphere is non-trivial, and storage can be high-dimensional. It is often used in theoretical analysis and for synthetic datasets in global illumination.

A Surface Light Field parameterizes rays based on their interaction with the explicit geometry of an object. Each ray is defined by a point x on the object's surface and an outgoing direction ω (typically in a local spherical coordinate system). The function is L(x, ω).

• Core Concept: Attaches the radiance function directly to an object's mesh or texture atlas. It separates view-dependent appearance (specular highlights, reflections) from base geometry.
• Efficiency: Highly compact for objects, as it only stores rays emanating from the surface, not all rays in space.
• Applications: Foundational for image-based rendering of real-world objects (e.g., the classic Unstructured Lumigraph) and a precursor to modern neural appearance models like Neural Radiance Fields (NeRF) which use a similar position-direction input.

The most fundamental parameterization represents a ray by its origin point o = (ox, oy, oz) in 3D space and its normalized direction vector d = (dx, dy, dz). This is the canonical form used in ray tracing and by coordinate-based neural networks like NeRF.

• Core Concept: Directly maps to the physical definition of a ray: r(t) = o + t * d, where t is the distance along the ray.
• Neural Rendering: This is the exact input format for a NeRF model: a multi-layer perceptron F(o, d) -> (c, σ) maps a 3D point and viewing direction to a color and volume density.
• Flexibility: Unconstrained by predefined planes or surfaces, making it ideal for volumetric scene representations. However, it lacks the structured, matrix-like organization of 2PP, making raw acquisition and certain processing operations more complex.

An Epipolar Plane Image (EPI) is not a separate parameterization but a powerful 2D slice and analytical tool derived from the 4D light field (typically under 2PP). By fixing one spatial dimension (e.g., v) and one angular dimension (e.g., t), you obtain a 2D image E(u, s).

• Core Concept: In this (u, s) slice, a scene point appears as a straight line. The slope of this line is inversely proportional to the depth of the corresponding 3D point.
• Primary Use: Enables highly efficient depth estimation and view interpolation through simple linear analysis in the EPI domain, without full 3D reconstruction.
• Visualization: EPIs provide an intuitive way to visualize the structure of a light field, where depth manifests as linear features, and occlusions create line discontinuities.

Concentric Mosaics represent a ray space captured by a camera moving along a circular path at a constant radius, typically pointing outward. Rays are parameterized by the camera's rotation angle θ, the vertical height v, and a horizontal pixel coordinate u on the camera's sensor.

• Core Concept: A 3D subset of the full 4D light field (L(θ, v, u)), designed for efficient rendering of panoramic scenes from a fixed viewpoint but with full horizontal parallax.
• Acquisition: Easier to capture than a full 4D light field, requiring only a simple camera rig on a circular rail.
• Rendering: Novel views within the circle are rendered by combining appropriate columns from different captured images, allowing a user to "look around" a central point. This was a key technique in early image-based rendering systems.

Ray space enables post-capture refocusing by synthetically shifting the integration plane for rays. After capturing a light field, the focal plane can be adjusted without any physical camera movement.

• Mechanism: Rays are re-integrated from the sensor to a virtual sensor plane at a different depth.
• Key Benefit: Enables 'focus after capture' for photography and computational microscopy.
• Example Tool: Lytro's first consumer light field camera demonstrated this capability.

By sampling rays from different angular coordinates, ray space allows the generation of novel viewpoints with correct parallax effects. This is the foundation for creating immersive 3D content and virtual camera walks.

• Core Process: Interpolating or extrapolating rays to render images from camera positions not physically captured.
• Applications: Free-viewpoint video, virtual reality content creation, and cinematic post-production.
• Constraint: Requires dense angular sampling to avoid artifacts like 'cardboarding'.

The slope of lines in Epipolar Plane Images (EPIs)—slices of ray space—directly encodes scene depth. Analyzing these linear structures provides highly accurate depth maps without active sensors.

• EPI Analysis: A point in 3D space manifests as a line in an EPI; the line's slope is inversely proportional to depth.
• Advantage: Provides dense, per-pixel depth estimation from passive capture.
• Use Case: Generating 3D models for digital twins, autonomous vehicle perception, and robotics.

Ray space allows for the angular filtering of light rays. By identifying and separating rays based on their direction of arrival, it is possible to computationally remove unwanted optical effects.

• Principle: Specular reflections and lens glare often occupy distinct angular regions compared to diffuse scene radiance.
• Process: Applying angular masks or filters in ray space to suppress rays from problematic directions.
• Impact: Improves image quality in computational photography and machine vision systems.

Ray space facilitates HDR capture from a single exposure. Different angular samples (sub-aperture images) can have varying effective exposures due to vignetting or lenslet design.

• Method: Merging these variably exposed samples reconstructs a scene with a wider dynamic range.
• Benefit: Mitigates motion blur artifacts common in multi-exposure HDR techniques.
• Application: Automotive imaging systems that must handle high-contrast roadside scenes.

Ray space is the native representation for light field displays. These displays emit a dense set of rays to different viewing positions, creating a 3D image viewable without glasses.

• Display Synthesis: The display hardware (e.g., lenslet arrays, layered LCDs) is designed to reconstruct a targeted ray space.
• Challenge: Requires extremely high spatial resolution to accommodate both spatial and angular dimensions.
• Future Direction: Enabling collaborative 3D visualization for design, medicine, and telepresence.

The plenoptic function is the complete theoretical description of all light in a scene, defined as a 7D function: P(θ, φ, λ, t, Vx, Vy, Vz). It represents the intensity of light observed from every 3D position (Vx, Vy, Vz), in every direction (θ, φ), for every wavelength (λ), at every time (t).

• Ray space is a practical, lower-dimensional parameterization of this function.
• It is the foundational concept from which all light field and image-based rendering techniques are derived.

A light field is a 4D or higher-dimensional subset of the plenoptic function that describes the radiance of light rays traveling in every direction through every point in a region of space. It is the core data structure for image-based rendering.

• Common representations use the two-plane parameterization, where a ray is defined by its intersections with two parallel planes (u,v and s,t). This is a direct instantiation of ray space.
• Captured by devices like plenoptic cameras or camera arrays, it enables computational effects like refocusing and novel view synthesis without explicit 3D geometry.

The two-plane parameterization is the most common method for defining a 4D light field in ray space. A unique light ray is parameterized by its intersection coordinates with two distinct, often parallel, planes in 3D space.

• The (u, v) coordinates on the first plane (often the aperture plane) define the ray's origin or direction.
• The (s, t) coordinates on the second plane (often the sensor/focal plane) define where the ray is sampled.
• The 4D coordinate (u, v, s, t) uniquely identifies a single ray, structuring the continuous light field into a discrete, tractable representation for processing and rendering.

A lumigraph is a specific, structured 4D data structure for representing and efficiently rendering a light field. It is closely related to ray space but incorporates an explicit 3D bounding geometry (like a cube or hull) around the scene.

• Rays are indexed by their intersections with this bounding geometry, which provides a scene-dependent parameterization.
• This structure allows for faster rendering and better handling of empty space compared to a raw, unorganized collection of rays.
• It represents an important engineering step from the pure mathematical concept of ray space to a practical rendering system.

An Epipolar Plane Image is a 2D slice extracted from a 4D light field (ray space) where one spatial dimension and one angular dimension are fixed. It is a powerful tool for analyzing light field structure and estimating depth.

• In an EPI, a static scene point appears as a straight line. The slope of this line is inversely proportional to the depth of the corresponding 3D point.
• This linear structure directly reveals the parallax encoded in the light field.
• EPI analysis is fundamental for many light field processing tasks, including depth estimation, super-resolution, and denoising.

Plenoptic sampling refers to the theory and practice of discretely capturing a continuous light field. It addresses the critical trade-offs between spatial and angular resolution governed by the plenoptic sampling theorem.

• For a fixed sensor, increasing angular sampling (more ray directions) reduces spatial sampling (resolution of each sub-aperture image).
• The sampling theorem defines the minimum rates required in both domains to avoid aliasing when reconstructing the continuous light field.
• This principle dictates the design of light field cameras and camera arrays, balancing the density of viewpoints against the detail in each view.