Real-Time Denoising

Real-time denoising algorithms are designed to work with severely under-sampled images, often with just 1-4 samples per pixel (spp). This is a fundamental constraint of interactive applications like games or VR, where full convergence via brute-force sampling is computationally impossible. The denoiser's primary task is to infer the converged, noise-free image from this sparse, noisy signal.

A core technique for stabilizing results and effectively increasing the sample count over time. The algorithm:

• Reprojects samples from previous frames onto the current frame using motion vectors.
• Accumulates color and variance information across multiple frames.
• Detects and rejects disoccluded pixels or pixels where reprojection fails (e.g., due to specular highlights) to prevent ghosting artifacts. This creates a stable, temporally coherent image sequence.

Denoising is not performed on the noisy color image alone. It relies on a set of auxiliary render passes, known as Arbitrary Output Variables (AOVs) or feature buffers, which provide essential geometric and material context. Critical buffers include:

• World Position & Normals: For spatial filtering and edge preservation.
• Albedo: Separates lighting from surface color to avoid blurring textures.
• Depth & Motion Vectors: For reprojection and depth-aware filtering.
• Roughness & Specular: To handle noisy highlights differently from diffuse surfaces.

The denoiser applies adaptive, non-linear filters in screen space. These are not simple blurs but edge-aware filters (e.g., cross-bilateral filters) that use the auxiliary buffers to guide smoothing. For example, the filter kernel will strongly smooth pixels that share similar depth, normal, and albedo, while preserving edges where these features change abruptly. Modern implementations often use wavelet transforms or non-local means filtering for higher quality.

The state-of-the-art approach replaces or augments hand-tuned filters with a convolutional neural network (CNN). The network is trained offline on massive datasets of noisy/clean image pairs. At runtime, it takes the noisy color and auxiliary buffers as input and outputs a denoised image. This allows it to learn complex noise patterns and scene priors, often achieving superior quality. NVIDIA's OptiX Denoiser and Real-Time Denoiser (NRD) are prominent examples, with some variants running in under 1 millisecond.

The defining constraint is the per-frame execution time, typically measured in milliseconds (e.g., <2 ms at 60 FPS). This necessitates highly optimized algorithms, often running on dedicated GPU hardware like tensor cores. Techniques include:

• Tiled rendering and dispatch to maximize parallelism.
• Low-precision inference (e.g., FP16 or INT8) for neural denoisers.
• Adaptive quality that reduces denoising effort for peripheral or motion-blurred regions. The entire pipeline, from buffer generation to final filter, must fit within this rigid budget.

Real-time denoisers operate by aggregating information across both space (neighboring pixels) and time (previous frames). This leverages temporal coherence, where most scene elements move predictably between frames. The process involves:

• Reprojection: Warping samples from previous frames to the current viewpoint using motion vectors.
• Temporal accumulation: Blending historical data with new samples to reduce variance.
• Spatial filtering: Applying edge-aware filters (e.g., a cross-bilateral filter) to the accumulated data to smooth noise while preserving sharp details like geometry edges and texture boundaries.

Real-time denoising is the enabling technology for interactive path tracing, allowing artists and developers to visualize photorealistic lighting with global illumination at interactive frame rates. The workflow is:

1. Sparse Sampling: The renderer casts only 1-4 rays per pixel per frame, generating a very noisy image.
2. Denoising: A spatiotemporal denoiser (like NRD) filters this input.
3. Real-Time Feedback: The clean result is displayed, allowing for immediate material and lighting adjustments. This is foundational in virtual production, architectural visualization, and next-generation game engines, replacing pre-baked lighting with dynamic, physically accurate results.

Accelerated NeRF frameworks like Instant Neural Graphics Primitives (Instant NGP) rely on real-time denoising to achieve interactive frame rates. The rendering process is inherently noisy due to:

• Low sample-count ray marching: To meet frame-time budgets, each ray takes few samples.
• Stochastic sampling: Importance sampling guided by a proposal network is non-deterministic. A real-time denoiser applied to the NeRF's raw output suppresses this Monte Carlo noise, enabling smooth, high-quality view synthesis for AR/VR and 3D content creation. This is a form of deferred neural rendering, where the neural field generates a noisy G-buffer that is then filtered.

Despite advances, significant engineering challenges remain:

• Disocclusion & Ghosting: When new geometry is revealed (disoccluded), historical data is invalid, causing ghosting artifacts. Solutions involve robust disocclusion detection and rejecting stale samples.
• Specular & Volumetric Effects: Highly view-dependent effects and participating media (fog, smoke) have poor temporal coherence, making them difficult to denoise.
• Hardware Integration: Future denoisers will be more tightly coupled with Neural Processing Units (NPUs) and use neural supersampling techniques to reconstruct high-resolution images from very low-resolution neural renders.
• Standardization: Efforts like Khronos' Open Denoising Standard aim to provide a vendor-agnostic API for denoising acceleration across hardware.

A Monte Carlo integration technique that reduces variance by concentrating computational samples in regions that contribute most to the final pixel color. In neural rendering, it is used to guide ray marching through a Neural Radiance Field (NeRF).

• Core Mechanism: Instead of sampling a ray uniformly, a probability distribution is learned or estimated to sample more points where density or radiance is high.
• Application: Drastically reduces the number of samples needed per ray, which is essential for achieving real-time performance. Often implemented using a lightweight proposal network.

A two-stage graphics pipeline that decouples geometry rasterization from neural shading to achieve real-time frame rates.

• Stage 1 (Rasterization): A traditional rasterizer produces a G-buffer, containing per-pixel geometric attributes like world position, normal, and material ID.
• Stage 2 (Neural Shading): A compact neural network processes the G-buffer to compute the final shaded color, enabling complex, view-dependent effects like neural textures and global illumination approximations.
• Benefit: The heavy neural network runs at screen resolution, not per 3D point, making it highly efficient for interactive applications.

A temporal accumulation technique that reuses color and motion data from previous frames to reduce noise, aliasing, and flickering in the current frame.

• How it Works: Uses per-pixel motion vectors to reproject samples from past frames onto the current frame. A history buffer blends these samples over time.
• Relation to Denoising: TAA is a foundational temporal denoiser. Modern real-time denoisers, especially for ray-traced effects, build upon TAA principles but use more sophisticated neural or analytical filters to handle extremely noisy, low-sample-count inputs.

A class of scene representations that combines an explicit data structure (e.g., a sparse voxel grid, hash table, or set of planes) with a small neural network for decoding.

• Examples: Instant NGP (multi-resolution hash grid), Plenoxels (sparse voxel grid), Tri-Plane Features.
• Advantage: The explicit structure allows for fast, O(1) lookup of features, while the tiny MLP decodes these into color and density. This balance is key for real-time inference, as it minimizes the computational burden of the neural network.

A deep learning-based spatial upscaling technique that renders a scene at a lower internal resolution and uses a neural network to reconstruct a high-resolution output.

• Process: The network takes the low-resolution render, along with auxiliary buffers like motion vectors and depth, and predicts a high-resolution image, filling in fine details.
• Relation to Performance: It is a complementary optimization to denoising. By rendering at a lower resolution, more compute budget can be allocated per pixel for sampling or denoising, which is then refined by the supersampling network. Technologies like NVIDIA DLSS are prominent examples.

A lightweight, auxiliary neural network used in volumetric rendering to predict an importance sampling distribution.

• Function: It evaluates a ray and outputs a set of intervals along the ray where the main scene representation (e.g., the NeRF) is likely to have non-zero density.
• Impact on Denoising: By guiding the primary renderer to sample more intelligently, the proposal network produces an initial, less-noisy estimate of the scene's geometry. This cleaner input significantly simplifies the task for any subsequent real-time denoising pass, improving both quality and performance.