Alpha Compositing

The over operator is the fundamental blending equation for alpha compositing. It defines how a foreground color C_f with opacity α_f is combined with a background color C_b. The resulting color C is calculated as:

C = α_f * C_f + (1 - α_f) * C_b

• This operation is associative but not commutative; order matters.
• In neural rendering, it is applied recursively along a ray from the camera into the scene.
• It mathematically approximates the physical process of light absorption and emission in a participating medium.

Alpha compositing provides a tractable, differentiable method to approximate the continuous volume rendering equation. The integral is discretized by sampling N points along a camera ray.

• At each sample point i, a neural network predicts a color c_i and a volume density σ_i.
• Density is converted to alpha α_i (transmittance) using: α_i = 1 - exp(-σ_i * δ_i), where δ_i is the distance to the next sample.
• The final pixel color is the weighted sum of all sample colors: C = Σ (T_i * α_i * c_i), where T_i is the cumulative transmittance.

A key characteristic is that the entire compositing pipeline is end-to-end differentiable. This enables gradient-based optimization of the underlying 3D scene representation.

• Gradients of a photometric loss (e.g., MSE between rendered and ground truth pixel colors) flow backward through the compositing equation.
• These gradients update the parameters of the neural network (e.g., a NeRF MLP) that predicts density and color.
• This allows the model to learn a coherent 3D representation from only 2D image supervision, without explicit 3D labels.

Alpha compositing inherently models order-dependent transparency. The final result is visually correct only when samples are composited in strict back-to-front order along each ray.

• This is because the over operator uses the accumulated transmittance (T_i) from all previous samples to weight the current one.
• In practice, samples are sorted by depth before compositing.
• This characteristic distinguishes volumetric alpha blending from order-independent transparency techniques used in traditional polygon rasterization.

Efficient alpha compositing relies on intelligent sampling along rays. Naive uniform sampling is computationally wasteful.

• Stratified Sampling: Initially samples points uniformly randomly within bins along the ray to prevent aliasing.
• Hierarchical Sampling (Importance Sampling): A two-pass process used in models like NeRF. A coarse network predicts a density distribution, which then guides a second, fine network to sample more points in high-density regions (likely containing surfaces).
• This focuses computation on parts of the volume that contribute most to the final pixel color, dramatically improving quality and efficiency.

In the context of implicit surface representations like Neural SDFs, alpha compositing is adapted to render surfaces defined by a zero-level set.

• Instead of a density field σ, a Signed Distance Function (SDF) f(p) is used.
• The SDF value is transformed into a weight or density using a logistic function (e.g., the sigmoid), creating a soft, volumetric "shell" around the surface.
• Alpha compositing then blends colors through this shell, allowing for the reconstruction of sharp, watertight surfaces while maintaining differentiability. This bridges volumetric rendering with surface rendering.

The foundational model that popularized alpha compositing for novel view synthesis. A NeRF represents a scene as a continuous 5D radiance field (3D location + 2D viewing direction). During rendering, differentiable ray marching samples points along a camera ray. At each sample, a small MLP predicts a RGB color and a volume density (sigma). The final pixel color is computed via the volume rendering integral, approximated using alpha compositing: colors are blended based on accumulated transmittance and the density at each point. This enables photorealistic reconstruction from sparse 2D images.

Models like NeuS adapt alpha compositing for high-quality surface reconstruction from Signed Distance Functions (SDFs). Instead of a density field, a neural network predicts an SDF value. NeuS derives a weighting function from the SDF that is used in the compositing equation. This function assigns maximum weight to points near the zero-level set (the surface). The alpha compositing process then blends colors from samples, but the weights are designed to be unbiased and surface-focused, leading to sharper geometry and less "haziness" compared to standard volumetric NeRF rendering.

Plenoxels (Plenoptic Voxels) represent a scene with an explicit sparse voxel grid storing spherical harmonic coefficients for color and a density value per voxel. Rendering involves ray tracing through this grid. Alpha compositing is performed directly on the voxel attributes without a neural network forward pass at each sample, making it extremely fast. The compositing equation remains the same: accumulate color and opacity along the ray. This demonstrates alpha compositing's role beyond purely neural representations, serving as the core renderer for explicit volumetric data structures.

Models for dynamic scenes, such as D-NeRF, extend alpha compositing to 4D (3D space + time). They learn a time-dependent deformation field and a canonical radiance field. For a given frame, rays are marched in canonical space. The alpha compositing integral now also depends on the temporal deformation. The core compositing logic—blending colors based on accumulated transmittance—remains unchanged, but the density and color predictions are conditioned on both spatial location and time, allowing for the rendering of realistic non-rigid scene motion from video.

The Volume Rendering Equation is the fundamental continuous integral that alpha compositing discretely approximates. It describes the physics of light transport through a participating medium (like fog or a neural density field), accounting for absorption and in-scattering along a ray. In NeRF, the neural network learns to represent the density and radiance fields that are plugged into this equation to synthesize novel views.

Differentiable Volumetric Rendering is the framework that makes the alpha compositing process trainable via gradient descent. It ensures the gradient of the photometric loss (difference between rendered and ground-truth pixels) can be backpropagated through the compositing steps to update the neural scene representation's parameters (e.g., density and color). This is the engine that allows NeRF models to be optimized from 2D images alone.

A Density Field (or Opacity Field) is a key volumetric component in neural rendering. It assigns a sigma (σ) value to every 3D point, representing the differential probability of a ray terminating at that location. During alpha compositing, this density is converted to alpha (α), the discrete probability of light being blocked at a sample, which directly controls the blending weights for color accumulation.

Differentiable Ray Marching is the specific numerical algorithm that implements alpha compositing for neural fields. It involves:

• Casting a ray from the camera through each pixel.
• Sampling 3D points along the ray.
• Querying the neural network for density and color at each point.
• Accumulating colors using the over operator. The entire sampling and accumulation pipeline is designed to be differentiable, enabling end-to-end optimization.

The Over Operator (or Porter-Duff over operator) is the specific compositing equation used in alpha blending. For two layers with colors C₁, C₂ and alphas α₁, α₂, the over operation is: C = α₁C₁ + (1 - α₁)α₂C₂. In volume rendering, this is applied recursively along a ray. The transmittance term (1 - α₁) represents the probability that light reaches subsequent samples, making the order of composition front-to-back critical.

Transmittance, denoted as T(t), is the probability that a ray travels from the camera to a depth t without hitting any particle. In the discrete alpha compositing approximation, it is calculated as the cumulative product of (1 - αᵢ) for all samples i before the current one. It acts as a weighting factor, ensuring that colors from occluded regions (where transmittance is near zero) do not contribute to the final pixel color.