Simultaneous Localization and Mapping (SLAM)

SLAM systems rarely rely on a single sensor. Sensor fusion combines data from multiple sources—such as monocular/stereo cameras, Inertial Measurement Units (IMUs), LiDAR, and wheel encoders—to create a robust state estimate. This redundancy is critical:

• Cameras provide rich visual features but suffer from motion blur and low-light conditions.
• IMUs offer high-frequency acceleration and angular velocity data, bridging gaps between camera frames.
• Fusion algorithms, like the Kalman filter or its nonlinear variants (e.g., Extended Kalman Filter), probabilistically combine these streams to produce a more accurate and stable pose estimate than any single sensor could provide.

At its core, SLAM is an estimation problem under uncertainty. It models the robot's pose (position and orientation) and map landmarks as random variables with associated probability distributions. The system continuously:

• Predicts the next state based on motion models (e.g., from an IMU or wheel odometry).
• Updates this prediction by incorporating new sensor observations (e.g., seeing a known landmark). This Bayesian filtering approach explicitly accounts for sensor noise and motion drift. Modern systems often use non-Gaussian approximations like particle filters or graph-based optimization to handle complex, non-linear relationships and multi-modal distributions.

SLAM architectures are typically decomposed into two interconnected modules:

• The Front-end (Perception): Processes raw sensor data into constraints. This involves feature detection and matching (e.g., using ORB or SIFT descriptors), data association (determining which observation corresponds to which map landmark), and constructing relative pose measurements between frames.
• The Back-end (Optimization): Takes the constraints from the front-end and performs state estimation. Historically, this used EKF-SLAM, but modern graph-based SLAM is dominant. Here, poses and landmarks are nodes in a graph, and sensor measurements are edges. The back-end solves for the most likely configuration of all nodes by minimizing the error across all edges, a process called bundle adjustment or pose graph optimization.

A defining capability of SLAM versus pure odometry is loop closure. As a robot moves, small errors in pose estimation accumulate, causing drift that distorts the map. Loop closure is the process of recognizing a previously visited location. When detected:

• The system identifies a visual place recognition match between the current view and a past keyframe.
• It adds a new constraint (edge) to the pose graph connecting the current pose to the historical pose.
• The back-end optimization then distributes the correction across the entire trajectory and map, enforcing global consistency. This is often achieved using Bag-of-Words models or convolutional neural network descriptors for efficient image retrieval from a large map.

The choice of map representation dictates the system's capabilities and computational load. Common representations include:

• Sparse Feature Maps: Store only distinct, recognizable landmarks (3D points). Efficient for localization but insufficient for navigation or interaction. Used in systems like ORB-SLAM.
• Dense Maps: Represent geometry at a high resolution, often as a point cloud, voxel grid, or Signed Distance Function (SDF). Essential for obstacle avoidance and AR occlusion. More computationally expensive.
• Semantic Maps: Augment geometric data with object-level labels (e.g., 'chair', 'door') from semantic segmentation. Enables higher-level reasoning and task-oriented navigation.
• Hybrid Representations: Modern systems often use a sparse graph for global optimization and a local dense map for immediate perception.

SLAM must operate in real-time on often constrained hardware (e.g., mobile phones, robots). This demands careful engineering:

• Keyframing: Not every frame is added to the map. Only informative keyframes are selected, limiting map growth.
• Local vs. Global Optimization: Full bundle adjustment over the entire map is computationally heavy. Systems typically run a local optimization in real-time and a slower global optimization in a parallel thread.
• Scalable Optimization: As the map grows, naive optimization becomes intractable. Techniques like hierarchical pose graphs, submapping, and incremental solvers are used to maintain constant-time updates.
• Hardware Acceleration: Critical for dense SLAM, leveraging GPUs for parallel operations like TSDF fusion or NPUs for neural inference in learned SLAM approaches.

Visual-Inertial Odometry (VIO) is a core sensor fusion technique that combines data from a camera and an Inertial Measurement Unit (IMU) to estimate a device's 6-degree-of-freedom (6DoF) pose. It is often the front-end of a SLAM system, providing high-frequency, short-term motion estimates.

• Mechanism: The camera provides visual constraints, while the IMU offers high-rate acceleration and angular velocity data, smoothing motion estimates during periods of poor visual tracking (e.g., fast motion, blur).
• Role in SLAM: VIO provides the local pose estimates that are later optimized and integrated into the global map during the SLAM back-end optimization.
• Example: Apple's ARKit and Google's ARCore use VIO for their world tracking capabilities.

Loop closure is the critical process in SLAM where a system recognizes it has returned to a previously visited location. This detection is essential for correcting accumulated drift—the small errors in pose estimation that compound over time and distance.

• Impact: When a loop is closed, it creates a constraint in the pose graph, allowing a global optimization (like bundle adjustment) to correct all past poses and map points, ensuring global consistency.
• Methods: Recognition is typically achieved by matching current visual features against a database of past keyframes using techniques like bag-of-words or learned descriptors.
• Consequence: Without loop closure, a SLAM system's map would become increasingly distorted and unusable for long-term navigation.

Bundle adjustment is a non-linear optimization technique that is the mathematical backbone of the SLAM back-end. It simultaneously refines the 3D coordinates of scene points (structure) and the camera poses (motion) to minimize reprojection error—the difference between where a 3D point is projected and where it is actually observed in an image.

• Function: It solves for the most probable map and trajectory given all noisy sensor measurements.
• Scale: In large-scale SLAM, pose-graph optimization is often used as a more efficient, sparse alternative, where the 3D points are marginalized out, and only camera poses are optimized.
• Tools: Frameworks like g2o and Ceres Solver are commonly used to implement bundle adjustment and pose-graph optimization.

A pose graph is a sparse graphical model used to represent the optimization problem in long-term SLAM. It is a more scalable representation than full bundle adjustment for large environments.

• Structure: Nodes represent estimated robot poses (6DoF). Edges represent spatial constraints between poses, derived from sensor measurements (e.g., odometry from VIO) or loop closure detections.
• Optimization: The goal is to find the set of node poses that maximize the likelihood of all the edge constraints, correcting drift globally. This is known as pose-graph optimization.
• Advantage: By focusing only on poses and not individual 3D map points, the optimization problem remains tractable for mapping cities or large buildings.

Sensor fusion is the overarching paradigm of combining data from multiple, heterogeneous sensors to produce state estimates that are more accurate, complete, and robust than those from any single sensor. SLAM is a premier example of sensor fusion.

• Common Sensor Suites:
  • Visual (Camera): Provides rich texture and feature data.
  • Inertial (IMU): Provides high-frequency motion and gravity reference.
  • Depth (LiDAR/ToF/RGB-D): Provides direct 3D geometry.
  • Wheel Odometry: Provides proprioceptive motion data.
• Fusion Algorithms: Techniques like the Kalman Filter (and its non-linear variant, the Extended Kalman Filter) and Particle Filters are classical approaches, while modern systems often use optimization-based methods.

Scene understanding moves beyond geometric mapping (the 'M' in SLAM) to infer semantic meaning. It involves parsing the environment to identify objects, surfaces, their categories, and their inter-relationships.

• Integration with SLAM: Modern Semantic SLAM systems co-optimize geometric mapping with semantic labels. For example, knowing a surface is a 'wall' provides a strong planar constraint for the map.
• Key Technologies:
  • Semantic Segmentation: Classifies every pixel in an image (e.g., floor, chair, person).
  • Instance Segmentation: Identifies and separates individual objects.
  • Plane Detection: Finds large, flat surfaces like floors and tables.
• Application: This enables intelligent AR content placement (e.g., "put the virtual lamp on the real table") and robotic navigation (e.g., "navigate to the kitchen").