Scene Flow

Unlike sparse feature tracking, scene flow aims to estimate a motion vector for every 3D point in the observed scene. This results in a dense, continuous field where each vector has three components (dx, dy, dz) representing the point's displacement in 3D Cartesian space. This density is crucial for applications requiring complete environmental understanding, such as predicting the trajectory of all points on a moving vehicle or deforming object.

Scene flow must account for both rigid motion (where all points on an object move coherently, like a rotating wheel) and non-rigid motion (where points move independently, like a walking person or flowing water). This combination makes the estimation problem particularly challenging, as it requires disentangling the motion of multiple independently moving objects, each of which may be deforming.

Scene flow is the 3D analogue of 2D optical flow. While optical flow estimates apparent 2D pixel motion in the image plane, scene flow estimates true 3D motion in world coordinates.

• Optical Flow = Projection of Scene Flow: The 2D flow observed in an image is a perspective projection of the underlying 3D scene flow.
• Disambiguation: Scene flow resolves the inherent ambiguity in optical flow, where a large nearby object moving slowly can produce the same 2D flow as a small distant object moving quickly.

Estimating scene flow requires geometric information. Common sensor setups and input modalities include:

• Stereo/RGB-D Video: Paired images or RGB-D frames from time t and t+1.
• LiDAR Sequences: Sequential 3D point clouds.
• Multi-View Systems: Multiple synchronized cameras.

The algorithm must often perform joint estimation, solving for depth, 2D optical flow, and 3D scene flow simultaneously to leverage mutual constraints and improve accuracy.

Scene flow is a foundational perception signal for robotics and autonomous systems operating in dynamic environments. It directly enables:

• Motion Segmentation: Clustering points based on common 3D motion to identify independent objects.
• Collision Prediction: Forecasting future 3D positions of all scene elements.
• Ego-Motion Compensation: Separating the robot's own motion from the motion of other objects.
• Interaction Understanding: Inferring how objects are being manipulated or are interacting with each other.

Estimating dense scene flow is a highly ill-posed and computationally intensive optimization problem. The search space is vast (3D motion for millions of points), and constraints like brightness constancy from 2D flow do not directly apply in 3D. Modern approaches use:

• Deep Learning: End-to-end networks (e.g., FlowNet3D, HPLFlowNet) that learn priors over scene motion from data.
• Regularization: Imposing smoothness constraints on the 3D motion field to make the problem tractable.
• Point Cloud Registration: Techniques like Iterative Closest Point (ICP) can be extended to estimate motion between sequential point clouds.

Optical flow is the 2D apparent motion of pixels between consecutive video frames, representing the projection of 3D scene motion onto the image plane. It is the 2D counterpart to scene flow.

• Key Difference: While optical flow estimates 2D pixel displacement, scene flow estimates the full 3D motion vector of each point in world space.
• Foundation: Optical flow is often used as an input or constraint for estimating the more complex scene flow.
• Applications: Video compression, action recognition, and video stabilization.

Visual Odometry is the process of estimating a camera's ego-motion (its own position and orientation change) by analyzing the sequence of images it captures.

• Relationship to Scene Flow: VO focuses on the camera's motion, while scene flow describes the motion of all objects relative to the camera. Accurate VO can help disentangle ego-motion from independent object motion when estimating scene flow.
• Sensor Input: Typically uses monocular or stereo cameras.
• Core Function: A fundamental component for robot navigation and Simultaneous Localization and Mapping (SLAM).

3D object detection is the task of identifying, classifying, and localizing objects in 3D space, typically by predicting oriented 3D bounding boxes from sensor data like LiDAR point clouds or RGB-D images.

• Dynamic Context: For dynamic scenes, 3D object detection provides the initial localization of objects whose 3D motion (scene flow) must then be tracked over time.
• Output: Provides a discrete, object-centric understanding of a scene, complementing the dense, per-point motion field of scene flow.
• Primary Sensor: Heavily reliant on depth-sensing technologies like LiDAR.

A point cloud is a set of discrete data points in a 3D coordinate system, representing the external surface of objects or an environment. It is the primary data structure for representing 3D scenes from sensors like LiDAR.

• Scene Flow Representation: Scene flow is often estimated and represented as a vector field associated with each point in a temporally sequential pair of point clouds.
• Challenges: Estimating motion directly on unstructured, sparse point clouds is a complex research problem.
• Downstream Use: The output of scene flow estimation is often used to register or align sequential point clouds.

RGB-D sensing is the simultaneous capture of color (RGB) and per-pixel depth (D) information, typically from a single sensor like a structured light or time-of-flight camera.

• Primary Data Source: RGB-D sequences are a common and practical input for scene flow estimation algorithms, providing aligned geometry and texture in real-time.
• Advantage: Delivers a dense, pixel-aligned depth map, which simplifies the correspondence problem compared to using only RGB images or sparse LiDAR.
• Applications: Widely used in robotics, augmented reality, and interactive systems.

Multi-View Stereo is a computer vision technique that generates a dense 3D reconstruction (e.g., a point cloud or mesh) from multiple calibrated images of a static scene taken from known viewpoints.

• Static Assumption: Traditional MVS assumes a static scene. Dynamic scene flow estimation can be seen as a generalization that relaxes this assumption, estimating geometry and motion.
• Foundation: MVS principles for establishing correspondences across views are extended in scene flow to establish correspondences across both views and time.
• Output: Produces the 3D geometry that serves as the basis for calculating 3D motion vectors.