Refocusing

This method uses the full 4D light field captured by a plenoptic camera. Refocusing is achieved by integrating rays that converge at a synthetic focal plane in post-processing.

• Shearing: The core operation is a 2D shear of the 4D light field data in ray space. This angular shift realigns rays to originate from a new virtual aperture center.
• Reintegration: After shearing, rays are summed across angular dimensions, simulating the integration that would have occurred at the sensor if the scene had been focused at that plane.
• Flexibility: Enables continuous, focus-free capture; any focal plane can be chosen after the fact, and depth of field can be adjusted arbitrarily.

This technique uses a focal stack—a sequence of images captured at different focus distances. Refocusing selects or blends from this pre-captured set.

• Depth-from-Defocus: Algorithms analyze the relative blur (defocus) of each point in the stack to estimate a depth map.
• Image Fusion: For synthetic refocusing, pixels are pulled from the image in the stack where they appear sharpest, based on the estimated depth map and the desired focal plane.
• All-in-Focus Imaging: A related output is an extended depth of field image created by fusing the sharpest regions from all images in the stack.

This is the foundational algorithm for digital refocusing from a light field. It mathematically replicates the function of a traditional lens.

• Ray Re-projection: For a desired focal plane, each sub-aperture image (a view from a specific microlens) is shifted on the sensor plane.
• Shift Amount: The shift is proportional to the disparity of the point, which is derived from its depth and the geometry between the main lens and microlens array.
• Averaging: The shifted sub-aperture images are averaged, which integrates light as if it had passed through a lens focused at the new plane. This is equivalent to a shearing operation in the 4D light field representation.

A fundamental constraint in light field refocusing. A plenoptic camera's sensor has fixed pixels, which must be allocated to sample both spatial and angular information.

• Spatial Resolution: The number of distinct pixels describing a single view of the scene.
• Angular Resolution: The number of distinct ray directions sampled per spatial point (e.g., the number of microlenses behind a main lens pixel).
• The Trade-off: Increasing angular resolution (for smoother refocusing and viewpoint interpolation) reduces the spatial resolution of the final rendered image. This is defined by the plenoptic sampling theorem.

Refocusing principles enable advanced capabilities in robotics, microscopy, and computational displays.

• Microscopy: In light field microscopy, refocusing allows 3D volumetric imaging from a single snapshot, enabling rapid observation of living samples.
• Robotic Vision: Provides immediate depth cues and allows for post-capture focus on regions of interest for inspection or navigation.
• 3D Displays: Drives holographic stereogram and integral imaging displays, where emitting the correct light field from a screen creates a glasses-free 3D image that viewers can focus on naturally.

Practical refocusing systems face several technical challenges that can degrade output quality.

• Angular Aliasing: Insufficient angular sampling causes ghosting or repeating artifacts in refocused regions, especially for out-of-focus highlights.
• Occlusion Artifacts: Objects at the synthetic focal plane may reveal regions that were occluded in the captured light field, leading to disocclusion errors that require inpainting.
• Noise Amplification: The shift-and-add process can amplify sensor noise, particularly in dark or heavily blurred regions.
• Resolution Loss: As noted in the spatial-angular trade-off, the final refocused image has lower resolution than a conventional photo from a sensor of the same size.

The most direct application of refocusing is allowing photographers to synthetically adjust the focal plane after an image is captured. This is achieved by computationally integrating rays from the desired focal plane within the captured light field. Key implementations include:

• Lytro cameras which popularized consumer light field photography.
• Focal stack fusion, where a stack of images at different focus distances is captured and the in-focus regions are blended to create an image with extended depth of field.
• Software tools that allow selective focus and bokeh adjustment from a single light field capture, effectively simulating different aperture sizes.

Refocusing algorithms are intrinsically linked to depth. By analyzing which synthetic focus setting brings different image regions into sharpness, a depth map can be inferred. This technique, known as depth-from-defocus, uses the focal stack as input. Applications include:

• Microscopy: Determining the 3D structure of biological samples.
• Industrial inspection: Measuring the height and topography of manufactured components.
• Computational imaging for robotics: Providing dense depth information from passive sensors, complementing active systems like LiDAR.

Refocusing enables the creation of images where the entire scene is in sharp focus, beyond the physical limits of a camera lens. This is critical in fields where maximum detail is required across a deep scene. The process involves:

1. Capturing a focal stack.
2. Using a focus measure (e.g., variance of Laplacian) to identify the sharpest pixels at each spatial location across the stack.
3. Fusing these pixels into a single all-in-focus composite. This is essential for:

• Macro and microphotography where depth of field is extremely shallow.
• Document digitization to ensure text is legible across curled pages.
• Landscape astrophotography to keep both foreground and stars sharp.

By selectively integrating rays from the light field, refocusing can simulate the effect of using a lens with a different aperture size. This allows for:

• Synthetic shallow depth of field: Creating professional-looking bokeh from images captured with small-aperture smartphone cameras. The quality depends on the angular resolution of the light field.
• Synthetic wide aperture: Simulating a faster lens for low-light performance by integrating more light rays.
• Bokeh editing: Altering the shape and character of out-of-focus highlights after capture, a feature found in advanced computational photography modes on modern phones.

Refocusing is the core computational backend for light field displays and holographic stereograms. These displays emit different light rays in different directions, allowing viewers to see a 3D scene without glasses. The process requires:

• Rendering dozens to hundreds of sub-aperture images (different viewpoints) from the captured or synthesized light field.
• These images are then fed to the display's optical system (e.g., lenticular lens array, parallax barrier, or directional backlight).
• The display hardware acts as the inverse of a light field camera, reconstructing the light field in space. This technology is used in experimental volumetric displays and emerging 3D monitors for design and medical imaging.

In scientific imaging, refocusing enables techniques that overcome physical limitations of optical systems. Key methods include:

• Fourier ptychographic microscopy: Captures multiple low-resolution images with varied illumination angles, then uses iterative phase retrieval and synthetic refocusing to reconstruct a high-resolution, large field-of-view, and depth-resolved image.
• Light field microscopy: Uses a microlens array to capture 4D light field data from a microscope, enabling rapid 3D imaging of dynamic biological processes (e.g., neural activity in zebrafish) without mechanical scanning.
• Optical coherence tomography (OCT): While not light-field-based, it shares the mathematical concept of synthesizing focus from interferometric data to create depth-resolved cross-sectional images of tissue.

A light field is a vector function that describes the amount of light flowing in every direction through every point in space. It is a 4D or higher-dimensional representation (parameterized by position and direction) that encodes the full radiance information of a scene. Capturing a light field, rather than a single 2D image, enables post-capture computational manipulations like refocusing, changing perspective, and altering depth of field. It is a practical subset of the full plenoptic function.

A light field camera (or plenoptic camera) is a specialized imaging device designed to capture a light field in a single photographic snapshot. It places a microlens array between the main lens and the sensor. Each microlens samples the directional distribution of light, allowing the sensor to record both the intensity and the direction of arriving rays. This raw capture is called a light field image and is the primary data source for software-based digital refocusing without requiring a focal stack.

A focal stack is a sequence of conventional 2D images of the same static scene, each captured with the camera focused at a different distance. This is an alternative data source for refocusing. Algorithms analyze the variation in blur (defocus) across the stack to estimate a depth map and an all-in-focus image. Refocusing is then achieved by synthetically re-blurring the all-in-focus image based on a new chosen focal plane. Unlike light field capture, this method requires multiple exposures and cannot synthesize novel viewpoints.

Sub-aperture images are a set of 2D views extracted during the processing of a single light field capture. Each image corresponds to the scene as seen from a different, small portion of the camera's main aperture (lens).

• They represent a discrete angular sampling of the light field.
• They can be used to simulate parallax and small viewpoint shifts.
• In refocusing algorithms, these images are shifted and averaged based on the desired focal plane to synthesize an image with a new depth of field.

Depth from Defocus is a computer vision technique that estimates scene depth by analyzing the relative blurriness of objects across two or more images with different focus settings. It is a core principle behind refocusing from a focal stack. The amount of defocus blur for a point is a function of its depth relative to the focal plane and the camera's aperture. By modeling this blur kernel (often as a Point Spread Function), algorithms can invert the process to estimate depth and enable synthetic refocusing.

The plenoptic function is the complete theoretical representation of all visual information in a scene. It is a 7D function: P(θ, φ, λ, t, Vx, Vy, Vz) describing the light intensity for every wavelength (λ), at every time (t), from every viewpoint position (Vx,Vy,Vz), and in every direction (θ, φ). A light field is a static, 4D slice of this function. Refocusing is a direct manipulation of this function—mathematically, it is an integration of rays over a new set of directions that would pass through a synthetic aperture at a new focal distance.