Plenoptic Function

The full, ideal plenoptic function is a 7D construct: P(θ, φ, λ, t, Vx, Vy, Vz). For static scenes and monochromatic light, this simplifies to the foundational 5D function: P(θ, φ, Vx, Vy, Vz).

• Vx, Vy, Vz (3D): The spatial position of the observer or camera.
• θ, φ (2D): The viewing direction (azimuth and elevation).
• This 5D function outputs the radiance (light intensity) for that specific ray defined by position and direction. A NeRF approximates this 5D mapping using a neural network.

The λ dimension accounts for the wavelength of light, which the human visual system perceives as color. In practice, this continuous dimension is discretized.

• Computer graphics and NeRF models typically represent this as a 3-channel RGB output (red, green, blue).
• More advanced spectral rendering might use many wavelength samples or learned spectral bases to model effects like dispersion or accurate material properties under different lighting.

The t dimension represents time, extending the function to model scenes that change. This is critical for representing dynamic events, moving objects, or non-rigid deformations.

• In Dynamic NeRF or 4D Neural Fields, time becomes an additional input to the neural network: P(x, y, z, θ, φ, t).
• This allows for the synthesis of novel views at novel times, enabling applications in video synthesis, free-viewpoint video, and modeling of fluid or elastic materials.

The three spatial dimensions Vx, Vy, Vz define every possible observation point within the scene's volume. This is the core of the "field" concept.

• A NeRF queries this continuous 3D space by sampling points along camera rays.
• The function's value changes based on location due to occlusion (objects block light), geometry (surface location), and participating media (like fog or smoke).

The two angular dimensions θ (azimuth) and φ (elevation) specify the direction of the ray leaving a 3D point. This encodes view-dependent appearance.

• This is essential for modeling non-Lambertian surfaces where color changes with viewpoint, such as:
  • Specular highlights (shiny surfaces).
  • Reflections.
  • Refraction (transparent materials).
• In a standard NeRF, the viewing direction is input to the network's later layers to predict view-dependent RGB color.

The full 7D function is intractable. Real-world systems make deliberate reductions, trading completeness for computational feasibility.

• Light Field (4D): Fixes time, wavelength, and radial distance (e.g., on a plane), resulting in P(u, v, s, t), capturing all rays through two planes. Used in light field cameras.
• Lumigraph: A structured, discretized version of the 4D light field.
• NeRF (5D): Approximates P(x, y, z, θ, φ) for a static scene with RGB output, using a neural network as a compact, continuous interpolator.
• Environment Map (2D): Fixes the spatial position V (e.g., at a single point), leaving only θ, φ. This describes the full panorama visible from that single point.

A Light Field is a practical, lower-dimensional slice of the full plenoptic function. It represents the radiance of light rays traveling in every direction through every point in a region of space, typically parameterized as a 4D function (2D spatial position + 2D angular direction).

• Core Concept: Captures all the visual information required to render images from any viewpoint within its bounds.
• Applications: Used in light field cameras (like Lytro), which capture angular information to enable post-capture refocusing and perspective shifts.
• Relation to Plenoptic Function: A light field assumes radiance is constant along a ray in free space (no occlusion changes), making it a simplified, more tractable 5D subset (removing wavelength and time) of the 7D plenoptic function.

The Lumigraph is a discrete, structured approximation of the light field, designed for efficient rendering from captured imagery. It is a 4D data structure where light rays are parameterized by their intersections with two parallel planes (a uv plane for spatial location and an st plane for direction).

• Core Concept: A sampled representation of the plenoptic function/light field, stored as a 4D array of radiance values.
• Rendering: Novel views are generated by interpolating between the sampled rays in this 4D space.
• Key Difference: While theoretically similar to a light field, the term 'Lumigraph' often emphasizes the specific two-plane parameterization and the associated rendering algorithms for computer graphics.

In this context, a Ray is a one-dimensional path of light defined by an origin (a 3D point in space) and a direction vector. It is the fundamental geometric primitive used to sample the plenoptic function.

• Mathematical Definition: A ray r(t) = o + t*d, where o is the origin, d is the normalized direction, and t is the distance along the ray.
• Role in Rendering: The color of a pixel is computed by integrating the plenoptic function's radiance values along the corresponding camera ray, accounting for occlusions and scattering.
• Connection: Evaluating the plenoptic function P(x, y, z, θ, φ, λ, t) for all points along a ray with fixed (θ, φ) and t provides the complete light contribution for that viewing direction.

Radiance is the radiometric quantity that describes the amount of light traveling in a given direction, per unit area perpendicular to that direction, per unit solid angle. It is the core output value of the plenoptic function.

• Units: Watts per steradian per square meter (W·sr⁻¹·m⁻²).
• Key Property: In a vacuum, radiance is constant along a ray. This property is fundamental to light field theory and ray-tracing algorithms.
• In the Plenoptic Function: The function P(x, y, z, θ, φ, λ, t) outputs spectral radiance—the radiance at a specific wavelength λ. The color perceived by a sensor is an integral of this spectral radiance weighted by the sensor's spectral response.

View Synthesis is the computer vision and graphics task of generating a photorealistic image of a scene from a novel camera viewpoint, given a set of input images. It is the primary practical application driven by plenoptic function theory.

• Goal: To compute P(x, y, z, θ, φ) for new (x, y, z, θ, φ) tuples corresponding to a new virtual camera.
• Approaches: Methods range from direct light field interpolation to modern Neural Radiance Fields (NeRF), which use a neural network to implicitly approximate the plenoptic function.
• Challenge: The 'hard' problem is to synthesize views with correct parallax and occlusion effects, which requires reasoning about the full 5D structure, not just image blending.

A Neural Radiance Field (NeRF) is a deep learning model that parameterizes a continuous volumetric scene representation as a 5D neural field. It is a direct, learned approximation of the plenoptic function.

• Function: A multilayer perceptron (MLP) maps a 3D location (x, y, z) and 2D viewing direction (θ, φ) to a volume density σ and RGB color c.
• Connection to Plenoptic Function: The NeRF model F(x, y, z, θ, φ) → (c, σ) is a machine-learnable proxy for the plenoptic function P, where time and wavelength are collapsed into the RGB output.
• Training: It is optimized via differentiable volume rendering to minimize the error between synthesized and ground-truth images, effectively 'solving' for the plenoptic function from sparse observations.