Hybrid Renderer

The hybrid pipeline uses hardware-accelerated rasterization to determine primary visibility—which triangles are visible in each pixel. This is exceptionally fast for filling the screen with geometry and generating the foundational G-buffer. The G-buffer stores per-pixel surface attributes like:

• World position
• Surface normal
• Material albedo (base color)
• Depth This explicit geometry pass provides a stable, high-performance base for subsequent lighting calculations.

After rasterization, selective ray tracing is applied. Rays are traced from the G-buffer data to compute complex lighting interactions that are poorly suited to rasterization. This typically includes:

• Accurate shadows (ray-traced shadows)
• Reflections (especially for non-planar surfaces)
• Global illumination (indirect lighting and color bleeding)
• Ambient occlusion By tracing rays only where needed (e.g., from pixels containing reflective materials), the system balances visual fidelity with performance.

Hybrid renderers are fundamentally deferred shading architectures. The G-buffer decouples geometry processing from lighting. All lighting calculations—both rasterized and ray-traced—are performed in screen space using the data stored in the G-buffer. This allows:

• A single lighting pass to apply to all visible pixels, regardless of geometric complexity.
• Efficient combination of multiple light sources.
• A natural integration point for ray-traced contributions, which are resolved and combined in this deferred lighting stage.

To achieve real-time performance, hybrid renderers use a low number of rays per pixel (often 1), which produces a noisy image. A real-time denoising pass is critical. This uses a neural network or a hand-tuned filter (like SVGF or BMFR) to clean the ray-traced data. Temporal Anti-Aliasing (TAA) is used to accumulate samples across frames, reusing information from previous frames via reprojection and motion vectors. This temporal accumulation further reduces noise and stabilizes the image.

Hybrid rendering employs several adaptive strategies to maintain frame rate:

• Variable Rate Shading (VRS): Shades less important image regions (e.g., periphery, motion-blurred areas) at a lower rate.
• Foveated Rendering: In VR, renders the center of gaze at full quality while reducing detail in the periphery.
• Dynamic Resolution Scaling: Temporarily lowers rendering resolution during heavy GPU load.
• Ray Budget Management: Dynamically adjusts the number of rays cast per frame based on available GPU time.

The hybrid renderer splits the rendering workload between two pipelines:

• Rasterization handles primary visibility, filling the screen with geometry at extreme speed. It produces a G-buffer containing per-pixel surface data (position, normal, albedo).
• Ray Tracing is then used selectively to calculate complex lighting effects that are difficult for rasterization, such as accurate shadows, reflections, and indirect diffuse lighting (global illumination). This combination provides a pragmatic balance between the raw performance of rasterization and the physical accuracy of ray tracing.

Hybrid rendering is critical for AR/VR and digital twin applications that demand both high fidelity and high frame rates.

• In VR, techniques like foveated rendering (often powered by Variable Rate Shading - VRS) can be combined with hybrid pipelines, applying ray tracing only to the high-resolution foveal region.
• For digital twins of factories or cities, rasterization efficiently draws millions of static assets, while ray tracing adds realistic lighting and material interactions for training simulations and architectural review, bridging the gap to photorealism.

Hybrid renderers are evolving to incorporate neural networks in a deferred neural rendering pipeline. The process is:

1. The traditional hybrid pipeline rasterizes a scene to produce a comprehensive G-buffer.
2. A compact neural network (e.g., a small MLP) processes this G-buffer data per-pixel.
3. The network can perform tasks like:
   • Neural supersampling (e.g., DLSS, FSR) from a lower internal resolution.
   • Applying complex, learned neural textures for ultra-detailed material properties.
   • Real-time denoising of ray-traced signals. This creates a three-way hybrid of rasterization, ray tracing, and neural processing.

The long-term trajectory for hybrid renderers is the gradual expansion of the ray tracing workload. Path tracing (tracing multiple bounces per pixel) is becoming feasible in real-time.

• Cyberpunk 2077's "Overdrive Mode" is a landmark example, using full path tracing for all lighting, supported by intensive AI denoising.
• The hybrid model persists here: rasterization may still be used for primary visibility or certain effects, while path tracing handles the complete light transport. The role of neural rendering and denoising becomes even more critical to reconstruct clean images from the sparse samples that real-time budgets allow.

A two-stage pipeline central to many hybrid renderers. First, a traditional rasterizer generates a G-buffer containing per-pixel geometry data (position, normal, material ID). A second, lightweight neural network then processes this buffer to apply complex shading, global illumination, or denoising.

• Key Benefit: Decouples expensive neural evaluation from geometric complexity.
• Example: Rasterizing a complex mesh, then using a small MLP to compute realistic soft shadows and ambient occlusion from the G-buffer data.

A scene representation strategy that combines an explicit data structure (like a sparse voxel grid or hash table) with a small neural network. The explicit structure provides fast spatial lookup and coarse geometry, while the neural network encodes fine details and view-dependent effects.

• Core Idea: Balances the speed of explicit methods with the quality and compactness of neural representations.
• Technologies: This is the foundation of Instant Neural Graphics Primitives (Instant NGP) and its multi-resolution hash grid.

Algorithms that remove noise from a partially sampled image within a strict frame-time budget (e.g., <2ms). In hybrid rendering, a low-sample-count ray tracer provides noisy global illumination data, which a denoising neural network (like NVIDIA's NRD or AMD's FSR) cleans up in real-time.

• Inputs: Noisy beauty pass, G-buffer (normals, albedo, depth), and motion vectors.
• Output: A clean, final frame suitable for interactive applications.

A critical technique for hybrid renderer stability. It reuses color and sub-pixel data from previous frames to smooth jagged edges (anti-aliasing) and further reduce noise. It relies heavily on accurate motion vectors and reprojection to align past frame data with the current view.

• Function: Accumulates samples over time, effectively providing a higher sample count for shading and denoising.
• Challenge: Can cause ghosting artifacts if motion vectors are incorrect.

A GPU hardware feature that allows a hybrid renderer to shade different regions of the screen at different rates. It is a key enabler for foveated rendering in VR/AR.

• Application: The center of the gaze is shaded at 1x1 pixel rate for maximum quality, while the periphery is shaded at 2x2 or 4x4 pixel blocks.
• Performance Gain: Can reduce pixel shading work by 30-50% with minimal perceptual loss, freeing resources for neural or ray tracing passes.

The volumetric rendering core of neural methods like NeRF, optimized for speed. Ray marching steps along a ray to sample a neural field. A proposal network accelerates this by performing coarse-to-fine sampling.

• Process: A lightweight proposal network first predicts likely density regions. The main rendering network then performs importance sampling only in those regions.
• Result: Drastically reduces the number of expensive neural network queries per ray, enabling real-time frame rates.