Reprojection

TAA is the dominant modern anti-aliasing method, relying entirely on reprojection for temporal accumulation. It reuses color and motion vector data from previous frames to reconstruct a cleaner current frame.

• Core Mechanism: Each pixel's history is reprojected from past frames using motion vectors. The current sample is then blended with this history to smooth jagged edges (aliasing) and reduce shader noise.
• Critical Dependency: Accurate motion vectors are essential. Incorrect vectors cause ghosting artifacts, where trails of old pixels appear behind moving objects.
• Performance/Quality Trade-off: Enables high-quality anti-aliasing at a fraction of the cost of supersampling (SSAA), making it standard in real-time game engines like Unreal Engine and Unity.

In virtual and augmented reality, reprojection is critical for combating motion-to-photon latency, which can cause simulator sickness. Techniques like Asynchronous Time Warp (ATW) and Spacewarp are emergency measures when the application misses its frame deadline.

• Process: When a new frame isn't ready, the last completed frame is reprojected using the latest, high-frequency head-tracking data. This creates an intermediate frame aligned with the user's current pose.
• Artifact Management: This works well for rotational head movements but can cause disocclusion artifacts (smearing) during rapid translation, as previously hidden areas (disocclusions) have no historical pixel data.
• System Role: A core component of the Oculus (Meta) and SteamVR runtime compositors, ensuring a judder-free experience even under system load.

Reprojection enables rendering at lower internal resolutions and reconstructing a high-resolution output, a key performance optimization. NVIDIA's DLSS and AMD's FSR use advanced forms of this.

• Workflow: The scene is rendered at a base resolution (e.g., 1080p). Motion vectors reproject this data to match the target resolution (e.g., 4K). A neural network or spatial upscaler then fills in missing detail and resolves the new pixels.
• Benefit: Delivers near-native image quality while significantly reducing GPU shading workload, enabling higher frame rates or more complex effects.
• Contrast with Spatial Only: Pure spatial upscalers (like bilinear filtering) look blurry. Temporal reprojection provides access to multiple unique samples per pixel over time, reconstructing sharper detail.

In neural graphics pipelines like Deferred Neural Rendering, reprojection is used to maintain temporal coherence and provide context to neural networks.

• G-Buffer Reprojection: Intermediate geometry buffers (G-buffers) from previous frames are reprojected to the current view. This provides the neural shader network with temporally aligned inputs (normals, depth, albedo), reducing flickering and instability.
• Training Acceleration: During the training of neural radiance fields (NeRFs), reprojection of cached rays or features can accelerate convergence by providing multiple observational constraints for a 3D point.
• Dynamic Scene Handling: For dynamic neural radiance fields, a deformation field performs 4D reprojection, mapping points from a canonical space to their position at a given time for coherent novel-view synthesis of moving scenes.

Advanced frame generation technologies like NVIDIA DLSS 3 Frame Generation use reprojection as a first step to create entirely new frames, boosting perceived frame rates.

• Optical Flow Analysis: An AI-powered optical flow field estimates per-pixel motion between two real frames far more accurately than traditional motion vectors.
• Reprojection & Synthesis: The last real frame is reprojected along this flow to an intermediate time position. A convolutional neural network then synthesizes the new frame, filling in disoccluded regions and resolving artifacts.
• Application: Particularly effective in GPU-limited scenarios, as it works after the main render, but requires high base frame rates to minimize latency.

Reprojection works in tandem with perceptual optimizations like Foveated Rendering and Variable Rate Shading (VRS) to maximize performance where visual detail matters least.

• Viewport Alignment: In VR, the high-resolution foveal region is tied to the user's gaze. As the eye moves, the rendered foveal region from the previous frame must be reprojected to match the new gaze location before being re-rendered at high detail.
• Temporal Stability: Reprojection helps stabilize the transition between shading rates in VRS, reducing noticeable flickering or swimming artifacts in the peripheral, low-shade-rate regions by blending with historical data.
• System Integration: Essential for making high-resolution, wide-field-of-view VR headsets feasible on current hardware.