Semantic Scene Understanding

A unified visual recognition task that fuses semantic segmentation (classifying every pixel as 'road' or 'sky') with instance segmentation (distinguishing individual 'car A' from 'car B'). This provides a complete, dense pixel-level description of a scene, assigning a class label to every pixel and a unique instance ID to countable objects. Modern architectures like Mask2Former achieve this in a single unified pipeline, eliminating the need for separate models.

The process of building a hierarchical, structured representation of a 3D environment as a graph. Nodes represent spatial entities (e.g., 'building', 'room', 'table', 'mug'), and edges define their relationships (e.g., 'room contains table', 'mug sits on table'). This moves beyond flat object lists to a queryable, relational model that enables reasoning about space and object affordances.

Assigning actionable meaning to objects beyond their categorical label. Instead of just identifying a 'flat, horizontal surface', the system predicts its affordance: 'this surface is sittable', 'this object is graspable by its handle', or 'this region is traversable'. This is critical for embodied AI agents and robots that must interact with, not just observe, their environment.

The fusion of semantic cues from heterogeneous sensor modalities to resolve ambiguity. A LiDAR point cloud provides precise 3D geometry but no color or texture; a camera provides rich appearance but poor depth. Semantic scene understanding aligns these streams, projecting camera-based class predictions onto the LiDAR depth map to create a dense, semantically labeled 4D point cloud (x, y, z, class) that is robust to the failure modes of any single sensor.

Maintaining coherent object identities and classifications across a sequence of frames. A car occluded by a pedestrian must retain its identity and not be re-initialized as a new object when it reappears. This requires object-centric tracking combined with a short-term memory mechanism that propagates semantic labels and instance IDs over time, ensuring a stable world model for downstream planning and prediction tasks.

The ability to identify objects and regions using arbitrary natural language descriptions, not just a fixed, pre-defined set of classes. Powered by vision-language models like CLIP, this allows a system to respond to queries like 'find the exit sign that is partially obscured' or 'locate all objects that could be used to block a doorway' without retraining, enabling zero-shot generalization to novel concepts and instructions.

The algorithmic prerequisite for scene understanding. Sensor fusion combines data from disparate sources like LiDAR, radar, and cameras to create a unified, noise-resistant environmental model. Without robust fusion, semantic labeling operates on incomplete or conflicting data.

• Fuses complementary sensor modalities
• Provides the geometric foundation for semantic segmentation
• Resolves data association ambiguities before classification

A mid-level fusion architecture that combines sensor data after it has been processed into symbolic representations like bounding boxes and class labels. This directly feeds the semantic scene graph by providing discrete object hypotheses.

• Refines object identity, position, and velocity
• Reduces bandwidth compared to raw data fusion
• Enables modular, sensor-agnostic scene models

A probabilistic framework for dynamic environment modeling that recursively estimates the likelihood of each cell in a spatial grid being occupied, free, or unmapped. This provides the binary state prior for semantic classification of drivable space and obstacles.

• Uses Bayesian inference to fuse sequential observations
• Handles sensor noise and transient objects
• Forms the grid-based substrate for semantic mapping

The foundational computational problem of building a map of an unknown environment while tracking the agent's pose within it. Semantic SLAM extends this by incorporating classified objects as landmarks, creating a globally consistent, interpretable map.

• Provides the spatial reference frame for scene labels
• Enables loop closure using semantic object recognition
• Transforms raw geometry into a persistent world model

The critical process of determining which sensor measurements originate from which physical objects across time. Accurate data association prevents identity switches and ensures that semantic labels remain temporally coherent as objects move through a scene.

• Essential for multi-target tracking
• Uses techniques like JPDA and MHT
• Maintains label consistency across frames

A graphical model framework that represents the scene understanding problem as a bipartite graph of variables (poses, landmarks) and constraints (sensor measurements). Solving this graph via nonlinear least squares produces the maximum a posteriori estimate of the environment's state.

• Unifies geometric and semantic constraints
• Enables globally consistent map optimization
• Backend for modern SLAM and SfM pipelines