Scene Flow Estimation

Scene flow is defined as a dense 3D vector field. For every 3D point in the scene at time t, it estimates a 3D displacement vector (Δx, Δy, Δz) predicting its position at time t+1. This differs fundamentally from optical flow, which estimates 2D pixel motion on the image plane. The output is a per-point motion estimate in the scene's world coordinate system, not the camera's view.

• Dense: Estimates motion for all visible scene points, not just sparse features.
• Metric: Vectors represent real-world motion (e.g., meters/second), not pixel displacement.
• Foundation: This 3D field is the foundational output for downstream tasks like motion segmentation and trajectory prediction.

Estimating 3D motion from 2D observations is an inverse problem with no unique solution. A single 2D pixel motion can correspond to infinite possible 3D motions. This aperture problem is exacerbated in 3D. Modern methods resolve this ambiguity by incorporating strong priors and multi-view constraints.

Key constraints used:

• Smoothness Prior: Assumes neighboring 3D points have similar motion.
• Rigidity Prior: Assumes objects often move as rigid bodies.
• Photometric Consistency: The appearance of a 3D point should be consistent across views and time.
• Geometric Consistency: The estimated 3D structure must be consistent with multi-view geometry.

Scene flow cannot be estimated independently of 3D geometry. Accurate flow requires an accurate 3D model of the scene (depth or point cloud), and conversely, accurate geometry can be refined using estimated motion. This leads to joint optimization frameworks.

Modern approaches often solve for:

• Depth (geometry)
• Optical Flow (2D correspondence)
• Scene Flow (3D motion)

Simultaneously, where each task informs the others. For example, a Neural Scene Flow Field (NSFF) jointly learns a dynamic Neural Radiance Field (NeRF) and the 3D flow field from monocular video.

Scene flow is the 3D unification of optical flow (2D image motion) and depth estimation (3D position). Formally, given depth Z and optical flow (u,v), scene flow can be derived via perspective projection. However, errors in either input propagate. State-of-the-art methods are self-supervised, learning all three quantities from video without ground truth labels.

• Optical Flow: 2D apparent motion (u, v) on the image plane.
• Depth: Distance Z from the camera to a 3D point.
• Scene Flow: The 3D motion vector (U, V, W) that, when projected, explains the observed 2D flow and depth change.

Real-world scenes contain non-rigid (e.g., clothing, fluids) and articulated (e.g., humans, animals) motion. Advanced scene flow methods model this complexity beyond simple rigid assumptions.

Common modeling strategies:

• Deformation Fields: Learn a continuous function mapping points from a canonical space to observed frames (used in Deformable NeRF).
• Articulated Models: Use skinning weight networks to model joint-based motion like a skeletal rig.
• Piecewise Rigid Assumption: Segment the scene into components that move rigidly (rigid motion decomposition).
• Motion Priors: Incorporate statistical models of likely motions (e.g., human pose priors).

Scene flow is not an end in itself but a foundational representation for higher-level spatial AI. It provides the raw motion data required for:

• Dynamic Object Segmentation: Separating independently moving objects from the background.
• Motion Prediction: Forecasting future positions of vehicles, pedestrians, or robots.
• Collision Avoidance: For autonomous navigation in dynamic environments.
• 4D Reconstruction: Building temporally coherent models for dynamic view synthesis.
• Action Recognition: Understanding activities from 3D motion patterns.

In essence, scene flow transforms a sequence of 3D snapshots into a coherent 4D spatiotemporal model of the world.