Motion Vectors

A motion vector is a per-pixel or per-block 2D displacement vector that quantifies the screen-space movement of scene content between two consecutive video frames or rendered images. It answers the question: "Where did this pixel come from in the previous frame?"

• Primary Role: Enables temporal coherence by linking corresponding scene points across time.
• Representation: Typically stored as a two-channel image (Δx, Δy) with sub-pixel precision.
• Core Use: The foundational data for motion estimation, a key step in video compression (e.g., H.264/HEVC), Temporal Anti-Aliasing (TAA), and frame interpolation.

Motion vectors are generated through algorithms that analyze frame pairs. The two primary methodologies are:

• Block-Based Motion Estimation: Divides the current frame into macroblocks (e.g., 16x16 pixels). For each block, it searches the previous frame within a defined search window to find the most similar block, using metrics like Sum of Absolute Differences (SAD). This is the standard for video codecs.
• Dense Optical Flow: Calculates a motion vector for every pixel by assuming brightness constancy and solving for pixel-wise motion, often using deep learning models like RAFT (Recurrent All-Pairs Field Transforms). This yields higher accuracy but is more computationally expensive.

Trade-off: Block matching is fast and hardware-friendly; optical flow is precise but slower.

In real-time rendering, motion vectors are the most important input for Temporal Anti-Aliasing. TAA works by accumulating samples across multiple frames to smooth jagged edges.

• Process: The current frame's color is blended with the reprojected color from the previous frame. Motion vectors tell the shader exactly how to reposition (reproject) each previous-frame pixel into the current frame's coordinate space.
• Challenge: Must account for both camera motion and object motion. Incorrect vectors cause ghosting or smearing artifacts.
• Implementation: The graphics pipeline (e.g., in Unity URP/HDRP or Unreal Engine) renders a dedicated motion vector pass alongside the color and depth passes.

Motion vectors enable the synthesis of new, in-between visual data, powering two key upscaling techniques:

• Frame Interpolation (e.g., DLSS 3 Frame Generation): To create a new Frame A→B, motion vectors are used to warp Frame A and Frame B toward the intermediate time step. A neural network then synthesizes the final interpolated frame, resolving disocclusions and complex motion.
• Temporal Super-Resolution (e.g., DLSS 2, FSR 2, TAAU): Renders the scene at a lower internal resolution. Multiple consecutive low-res frames are aligned using their motion vectors and combined by a neural network or heuristic algorithm to reconstruct a high-resolution output. This provides a superior quality/performance trade-off than spatial upscaling alone.

The storage and precision of motion vectors are engineered for performance and quality.

• Precision: Modern renderers use at least 16-bit floating-point (half-float) per channel to store sub-pixel motion. Integer pixel motion is insufficient for high-quality temporal accumulation.
• Coordinate Spaces:
  • Screen-Space: Most common. Vectors are in 2D pixel/displacement units.
  • World-Space: Sometimes used for more stable reprojection under complex camera projection changes. Requires rendering a world-position buffer.
• Velocity Buffer: In graphics APIs, the rendered motion vector output is typically called a velocity buffer. This buffer must be carefully managed to handle motion from skinned animation, particle systems, and camera jitter.

Incorrect or missing motion vectors are a primary source of visual artifacts in temporal techniques.

• Common Artifacts:
  • Ghosting: Trailing remnants of fast-moving objects caused by incorrect vector assignment or blending over disoccluded regions.
  • Motion Blur Mismatch: If object motion blur in the color buffer doesn't match the sharp vectors in the velocity buffer, smearing can occur.
  • Under/Overflow: Vectors pointing outside the screen bounds must be handled.
• Mitigation Strategies:
  • Neighborhood Clamping: In TAA, the history sample is clamped to the color range of the current pixel's neighbors to reject outliers.
  • Vector Validation: Using depth and normal buffers to reject vectors that imply impossible scene motion (e.g., a vector connecting two unrelated surfaces).
  • Fallback Samples: Using spatial-only filtering when temporal data is deemed unreliable.

A post-processing technique that uses motion vectors and data from previous frames to smooth jagged edges (aliasing) and reduce noise. It works by:

• Reprojecting samples from past frames onto the current frame using motion vectors.
• Applying a temporal filter to blend history with the current frame.
• Using techniques like neighborhood clamping to reduce ghosting artifacts. It is the industry standard for anti-aliasing in real-time engines like Unreal Engine and Unity.

The core mathematical operation of transforming pixel data (color, depth) from one camera viewpoint to another. It is essential for:

• Temporal stability in TAA and motion blur.
• Virtual reality techniques like Asynchronous Time Warp (ATW) to reduce latency.
• Dynamic neural radiance fields for aligning observations across time. The process requires an accurate depth buffer and per-pixel motion vectors to handle disocclusions correctly.

A set of intermediate render targets storing per-pixel surface attributes for deferred shading pipelines. A standard G-buffer includes:

• World Position
• Surface Normal
• Albedo (base color)
• Roughness & Metallic (PBR material properties)
• Motion Vectors (screen-space velocity) This data structure is critical for deferred neural rendering, where a neural network processes the G-buffer to add complex effects like global illumination or super-resolution.

A 4D neural scene representation that models geometry, appearance, and motion over time. To model non-rigid deformation, these systems often employ:

• A canonical NeRF representing the scene in a rest pose.
• A deformation field network that maps 3D points from a given timestamp back to the canonical space.
• Motion vectors can be derived analytically from the deformation field's Jacobian, connecting learned 4D representations to traditional rasterization pipelines for hybrid rendering.

The process of generating a high-resolution frame by fusing information from multiple consecutive low-resolution frames. Advanced implementations:

• Use motion vectors to align sub-pixel details across frames accurately.
• Employ a neural network (e.g., NVIDIA DLSS, AMD FSR) to reconstruct high-frequency detail.
• This is a form of neural supersampling, allowing games to render at a lower internal resolution for performance, then upscale with minimal quality loss.

A virtual reality technique that reduces perceived latency and prevents judder. If a new frame from the application is delayed, the VR runtime:

• Takes the most recently completed frame.
• Reprojects it using the latest head-tracking data and stored motion vectors.
• Warps the image to match the user's current head pose. This generates an intermediate frame, maintaining smooth motion at the cost of potential positional artifacts during rapid movement.