Keyframe

A keyframe is a selectively chosen frame from a continuous sensor stream (e.g., camera, LiDAR) that meets specific criteria for long-term retention. It is not every frame.

• Selection Criteria: Frames are chosen based on metrics like sufficient parallax (camera motion), number of new observed landmarks, or information gain relative to existing keyframes.
• Purpose: This selection drastically reduces the number of states to optimize in the back-end, turning an intractable problem of optimizing every frame into a sparse, manageable one.
• Example: In ORB-SLAM, a new keyframe is created when tracking quality is good and the camera has moved a significant baseline from the last keyframe, ensuring new 3D points can be triangulated.

A keyframe serves as the primary observational anchor for a set of 3D landmarks (map points). It stores the descriptors and pixel coordinates of features associated with these landmarks.

• Data Association: The feature descriptors (e.g., ORB, SIFT) stored in the keyframe are used for matching against current frames for tracking and for loop closure detection against past keyframes.
• Observation Database: Each keyframe maintains a list of which 3D map points it observes and their 2D projections. This creates the essential projection factors used in bundle adjustment or graph optimization.
• Implication: Deleting a keyframe often requires culling or re-triangulating its associated map points, affecting map density.

In modern graph-based SLAM systems like g2o or GTSAM, each keyframe becomes a node in a pose graph or factor graph.

• Node State: The node contains the 6-DoF pose (position and orientation) of the sensor at the time the keyframe was captured.
• Constraint Edges: Edges between keyframe nodes represent spatial constraints. These can be from odometry (between consecutive keyframes), loop closures (between non-consecutive keyframes), or relative pose measurements from sensor fusion.
• Optimization: The back-end optimization solves for the most probable configuration of all keyframe poses given this network of constraints, minimizing drift.

The core rationale for keyframes is to manage computational and memory resources in long-term, large-scale SLAM.

• Memory Efficiency: Storing full sensor data for every frame is prohibitive. Keyframes provide a sparse yet information-rich summary of the trajectory and environment.
• Optimization Scalability: Bundle adjustment complexity grows non-linearly with the number of frames. By limiting optimization to keyframes, real-time performance is maintained over extended operations.
• Culling Strategies: Systems implement keyframe culling policies to prevent unbounded growth. A keyframe may be removed if over 90% of its map points are seen in three other keyframes, ensuring sparsity without information loss.

Keyframes are inherently sparse in time but dense in information. Their distribution along the trajectory is non-uniform and adaptive.

• Temporal Sparsity: Keyframes are not captured at fixed time intervals. The rate adapts to motion and scene change. In fast motion, keyframes may be created more frequently to maintain tracking.
• Spatial Coverage: The set of keyframes should provide overlapping observational coverage of the environment to support robust triangulation and loop closure. The baseline between keyframes is critical for accurate 3D reconstruction.
• Metric: A common measure is the median scene depth or the parallax angle to previous keyframes to decide creation.

Keyframes are the fundamental unit for place recognition and loop closure, which are essential for correcting accumulated drift.

• Query Database: When the system enters a previously visited area, the current frame is matched against a database of keyframe descriptors (often using a Bag-of-Words model like DBoW2) to find candidate matches.
• Geometric Verification: A candidate loop closure keyframe undergoes rigorous geometric verification (e.g., using RANSAC with a fundamental or homography matrix) to reject false positives.
• Constraint Addition: A validated loop closure creates a new constraint edge in the pose graph between the current keyframe and the past keyframe, triggering a global optimization to correct the entire map and trajectory.

A non-linear optimization technique that jointly refines the estimated 3D structure of a scene (landmarks) and the camera poses (keyframes) by minimizing the total reprojection error across all observations. It is the computational core that uses keyframes to produce a globally consistent map.

• Function: Takes keyframes and their associated feature observations as input.
• Output: Optimized 3D landmark positions and refined keyframe poses.
• Scale: Typically run as a local optimization over a window of recent keyframes or as a global optimization during loop closure.

A simplified graph representation used in Graph SLAM where nodes represent robot poses (often corresponding to keyframes) and edges represent spatial constraints between them. Keyframes become the nodes in this graph.

• Nodes: Each keyframe's estimated position and orientation (pose).
• Edges: Constraints derived from sensor measurements between keyframes (e.g., from visual odometry or loop closure).
• Optimization: The back-end adjusts all node poses to satisfy the edge constraints, correcting drift. This makes the collection of keyframe poses globally consistent.

The real-time perceptual pipeline that processes raw sensor data to create candidate keyframes and initial pose estimates. It decides when to create a new keyframe.

Key tasks include:

• Feature Extraction & Tracking: Identifying distinctive points (e.g., ORB features) across image frames.
• Initial Pose Estimation: Calculating motion between frames using algorithms like RANSAC.
• Keyframe Decision Logic: Heuristics based on traveled distance, rotational change, or tracking quality determine if the current frame is distinctive enough to be promoted to a keyframe.

The detection that the agent has returned to a previously visited location, followed by a correction of accumulated drift. It is the primary mechanism for enforcing long-term map consistency using keyframes.

• Detection: A place recognition module matches the current view against a database of past keyframes.
• Constraint Creation: Upon a positive match, a new spatial constraint is added between the current keyframe and the matched historical keyframe.
• Optimization Trigger: This new constraint is fed into the back-end optimization (e.g., the pose graph), which redistributes the error and corrects the drift in all keyframe poses and the map.

The process of estimating the incremental ego-motion of a camera by analyzing the apparent motion of visual features between consecutive frames. It provides the short-term motion estimate between keyframes.

• Function: Provides the relative pose constraints that connect sequential keyframes in the pose graph.
• Drift: VO estimates accumulate error over time (drift).
• Role for Keyframes: Keyframes act as anchors; VO operates between them, but the absolute position of each keyframe is corrected during global optimization (bundle adjustment, pose graph).

A graph representation used within SLAM systems where nodes are keyframes and edges connect keyframes that observe common 3D map points. It defines local neighborhoods for efficient operations.

• Structure: An edge exists between two keyframes if they share a sufficient number of observed landmarks.
• Uses:
  • Local Bundle Adjustment: Optimization is performed over a keyframe and its covisible neighbors.
  • Data Association: Limits search for feature matching to related keyframes.
  • Redundancy Management: Helps identify and cull redundant keyframes.