Occlusion Handling

This foundational technique reconstructs a depth map for each input view. Occlusions are identified where a 3D point projects into one view but is occluded by a closer surface in another. The core algorithm involves:

• Cost volume construction aggregating matching costs across hypothesized depths.
• Winner-takes-all or global optimization to select the best depth per pixel.
• Visibility reasoning to invalidate pixels where the projected depth conflicts with other views. Common challenges include depth discontinuities and textureless regions where matching fails.

NeRF uses a continuous 5D neural scene representation (3D location + 2D viewing direction). It handles occlusions implicitly through volume rendering, which integrates color and density along a camera ray. Key mechanisms are:

• The density field σ determines where light is absorbed or scattered; high density blocks rays.
• The transmittance function T(t) models the probability a ray travels to depth t without hitting anything.
• The final pixel color is a weighted sum: C = Σ T(t) * σ(t) * c(t). This naturally models semi-transparent and complex occluding geometry without explicit masks.

A geometric approach leveraging epipolar geometry. For a pixel in a source image, its corresponding point in a target view must lie along a corresponding epipolar line. Occlusion detection proceeds by:

• Searching along the epipolar line for a matching feature or color.
• Identifying occlusion regions where no consistent match exists because the point is hidden.
• Using ordering constraints (e.g., the order of surfaces along a scanline is consistent between views). This is fundamental to stereo matching and light field processing, where occlusions manifest as discontinuities in the disparity field.

This technique explicitly represents occlusions using multiple depth layers. A Layered Depth Image (LDI) stores, for each pixel, a list of surfaces sorted by depth. Rendering a novel view involves:

• Warping each layer to the new viewpoint.
• Alpha compositing layers from back to front using their associated colors and opacities.
• Z-buffering to resolve visibility conflicts within and between layers. This is highly effective for dynamic scenes and real-time rendering, as used in some image-based rendering pipelines. It handles thin structures and translucent materials well.

For video or dynamic scenes, occlusions are tracked over time. This involves estimating optical flow (motion between frames) and detecting occlusion regions where flow is invalid or unreliable. Techniques include:

• Forward-backward consistency checks: A point visible in frame t should map to the same point when warped to t+1 and back to t. Large errors indicate occlusion.
• Inpainting: Using information from adjacent frames or spatial neighbors to fill occluded regions in synthesized views.
• Motion-in-depth reasoning: Distinguishing between disocclusion (newly revealed areas) and occlusion (newly covered areas) based on 3D motion.

Modern deep learning approaches train neural networks to directly predict visibility masks or soft visibility weights. These models learn from data to reason about complex occlusion patterns. Common architectures include:

• Cost volume networks that learn to aggregate multi-view information and output a visibility-aware depth or disparity.
• Transformer-based models that use attention mechanisms to match features across views, inherently learning which views provide valid information for each 3D point.
• Generative models like conditional GANs that can hallucinate plausible content for disoccluded regions. These methods excel with unstructured image collections but require significant training data.

Multi-view stereo (MVS) is a core 3D reconstruction technique that generates dense geometry from overlapping 2D images. It directly confronts the occlusion problem:

• Goal: Estimate a depth or 3D point for every pixel in a reference image.
• Occlusion Challenge: A 3D point may be visible in the reference view but hidden in other source views, creating invalid photo-consistency matches.
• Standard Solution: MVS algorithms use visibility reasoning to identify and exclude occluded source views from the matching cost calculation for each candidate 3D point.

Photo-consistency is the fundamental constraint used to verify 3D geometry in multi-view reconstruction. It states that a correctly estimated 3D point, when projected into all images where it is visible, should have similar color or intensity values.

• Occlusion's Role: Occlusions violate this assumption. A point projected into a view where it is occluded will match against the occluder's color, not its own, leading to erroneous geometry.
• Handling: Robust photo-consistency measures (e.g., normalized cross-correlation over patches) and explicit occlusion detection are used to down-weight or ignore inconsistent projections.

Depth map fusion is the process of merging per-view depth estimates (from MVS) into a unified, global 3D model (e.g., a point cloud or mesh). Occlusion handling is critical here:

• Problem: Depth maps from different viewpoints contain conflicting information in occluded regions—one map shows the occluder, another shows a gap.
• Solution: Fusion algorithms like Volumetric or Surface-based integration use visibility voting. A 3D voxel or point is only confirmed if it is visible and consistent across a sufficient number of input views, filtering out ghost geometry from occluded areas.

View-dependent appearance refers to how the color or radiance of a 3D point changes based on the viewing direction, due to effects like specular highlights or non-Lambertian surfaces. This is often conflated with occlusion.

• Distinction: A point may appear different in two views not because it's occluded, but due to a change in reflection.
• Advanced Handling: Modern neural methods (like NeRF) model this explicitly with a view-dependent radiance component. This allows the system to disambiguate true occlusions (where no information exists) from legitimate appearance changes.

Ray marching is the iterative process used in volume rendering and neural fields (NeRF) to accumulate color and density along camera rays. Efficient occlusion reasoning is inherent to this process.

• Mechanism: As a ray marches through space, it samples volume density (opacity). High-density regions block (occlude) samples behind them.
• Neural Rendering: In a Neural Radiance Field (NeRF), the network predicts density σ and color c at each sample point. The rendering integral automatically accounts for occlusions by attenuating the contribution of downstream samples based on accumulated transmittance.

Alpha compositing (or over operator) is the digital image compositing technique used in volume rendering to blend samples along a ray, providing a natural mathematical model for partial occlusion.

• Formula: C = Σ (Ti * αi * ci), where αi is opacity (from density), ci is color, and Ti is the accumulated transmittance (how much light reaches that sample).
• Role in Occlusion: Each sample with α > 0 occludes samples behind it by reducing Ti. This provides a continuous, differentiable model for handling complex occlusions like semi-transparent objects or fuzzy boundaries.