Differentiable Shading

A BRDF is a mathematical function that defines how light is reflected at an opaque surface, given the incoming and outgoing light directions. It is the core model in physically based rendering (PBR). For differentiability, common analytic BRDF models like the Cook-Torrance (for metals/rough surfaces) or Disney BRDF (for artist-friendly parameters) are used. Their parameters—albedo, roughness, metallic, and specular—are made continuous so gradients can flow through them, allowing a system to learn, for example, that increasing roughness reduces the intensity of a specular highlight.

This refers to the derivative of the rendered pixel color with respect to the underlying material properties. During optimization, these gradients answer: 'How should I change the albedo or roughness to make this pixel match the target image?'

• Albedo Gradient: Indicates how to adjust the base color/diffuse reflectance.
• Roughness Gradient: Guides adjustments to surface microfacets, blurring or sharpening reflections.
• Normal Gradient: Though related to geometry, shading is highly sensitive to the surface normal, which defines the local orientation for lighting calculations. Differentiable shading provides gradients to optimize these normals.

Differentiable shading calculates how changes in light direction, intensity, or environment map values affect the final image. This enables inverse lighting—recovering illumination from a photo.

• Direct Illumination: Gradients show how moving a light source changes shading and cast shadows.
• Environment Map Optimization: The colors of an HDR environment map can be optimized to match the reflections and diffuse lighting observed in multiple images.
• Soft Shadows: Gradients can even propagate through shadowing functions, allowing the system to reason about occluder geometry based on soft shadow penumbras.

An SVBRDF is a texture map where each texel stores a full set of BRDF parameters (albedo, roughness, etc.), allowing complex materials like wood or fabric. Differentiable shading optimizes these high-dimensional texture maps. The process often involves:

• Differentiable Texture Sampling: Using continuous, sub-texel coordinate lookups so gradients can flow back to update the texture values.
• Multi-Image Optimization: Using many photos under different lighting/viewing conditions to disentangle and uniquely solve for the SVBRDF at each point on the surface.

Differentiable shading is typically the final stage in a differentiable rasterization pipeline (e.g., Soft Rasterizer, Neural Mesh Renderer). The pipeline first determines which triangles are visible and their barycentric coordinates at each pixel (a differentiable process), then passes those coordinates to the shading stage. The shading computation—applying the BRDF using interpolated normals and material parameters—is inherently differentiable, completing the gradient path from the final image pixels back to the 3D mesh vertices and texture maps.

The primary application of differentiable shading is inverse rendering. By minimizing a photometric loss between rendered and real images, systems can automatically recover:

• High-Quality Material Assets: From casual photo collections, enabling rapid 3D content creation.
• Relighting: Objects can be seamlessly inserted into new environments with correct lighting.
• Digital Twins: Creating physically accurate models of real-world objects for simulation. Frameworks like nvdiffrec and PyTorch3D implement these full pipelines, combining differentiable rasterization with differentiable shading to solve for geometry, materials, and lighting jointly.

Differentiable rendering is a class of computer graphics techniques that make the entire image synthesis pipeline differentiable. This enables the calculation of gradients with respect to scene parameters—such as geometry, materials, and lighting—allowing for gradient-based optimization. It is the overarching framework within which differentiable shading operates.

• Core Mechanism: It reformulates traditional, non-differentiable operations (like visibility testing and rasterization) using smooth approximations or analytic derivatives.
• Primary Application: Inverse graphics, where the goal is to infer 3D scene properties from 2D images by minimizing a rendering loss.

A Bidirectional Reflectance Distribution Function (BRDF) is a mathematical function that defines how light is reflected at an opaque surface. It is the fundamental model that differentiable shading techniques make differentiable.

• Inputs: Incoming light direction and outgoing view direction.
• Outputs: The ratio of reflected radiance to incident irradiance.
• Common Differentiable Models: The Phong and Cook-Torrance models are frequently used. Making these models differentiable involves computing partial derivatives with respect to parameters like surface normal, roughness, and albedo.

A material gradient is the derivative of a rendered pixel's color (or a full image's loss) with respect to the parameters of a surface material model. It is the direct output of applying differentiable shading.

• What it Quantifies: How a small change in a material property—like base color, metallicness, or roughness—affects the final rendered appearance.
• Use in Optimization: During inverse rendering, these gradients are backpropagated to update material parameters, guiding them toward values that minimize the difference between a rendered image and a target photograph.

Inverse graphics (or inverse rendering) is the process of estimating the underlying 3D scene properties from 2D observations. Differentiable shading is a critical enabling technology for this task.

• Classical Approach: Traditionally involved complex, non-convex optimization without gradient guidance.
• Modern, Gradient-Based Approach: Uses a differentiable renderer (including a differentiable shader) to compute gradients of a photometric loss, enabling efficient optimization via gradient descent. This allows joint estimation of geometry, materials, and lighting from images.

Automatic differentiation (Autodiff) is a set of techniques for automatically and efficiently computing derivatives of functions defined by computer programs. It is the computational engine that makes differentiable shading practical.

• How it Works: It breaks down the shading function into a graph of elementary operations (add, multiply, sin, etc.) and applies the chain rule systematically.
• Modes: Forward-mode and reverse-mode (backpropagation). Modern deep learning frameworks like PyTorch and JAX use reverse-mode autodiff, which is highly efficient for functions with many input parameters (like scene properties) and a scalar output (like loss).

SVBRDF optimization is the process of estimating a Spatially-Varying Bidirectional Reflectance Distribution Function—a texture map of material parameters—using differentiable rendering. It is a direct and advanced application of differentiable shading.

• Goal: Recover a full material map (albedo, normal, roughness, etc.) for a complex object from a set of photographs.
• Process: A differentiable renderer, with a differentiable shading model, renders the object under known or estimated lighting. The gradients of the difference between these renders and the input photos are used to update the SVBRDF texture maps.