Semantic Segmentation

Unlike object detection which draws bounding boxes, semantic segmentation assigns a class label (e.g., 'road', 'pedestrian', 'building') to every individual pixel in an image or point in a 3D scan. This creates a dense, detailed understanding of scene composition, essential for tasks requiring precise boundaries, such as autonomous vehicle path planning or robotic manipulation.

• Output: A segmentation mask where each pixel's color corresponds to a class.
• Granularity: Enables differentiation between adjacent objects of the same class (e.g., two separate cars) based on pixel grouping, though it does not distinguish individual instances (see Instance Segmentation).

It's crucial to distinguish semantic segmentation from related tasks:

• Semantic Segmentation: Labels every pixel with a class (e.g., 'car').
• Instance Segmentation: Identifies and delineates each distinct object, assigning a unique label to all pixels of each individual instance (e.g., 'car-1', 'car-2').
• Panoptic Segmentation: A unified task that combines both, labeling all pixels with a stuff class (amorphous regions like 'sky', 'road') or a thing class with an instance ID (countable objects like 'car'). This provides the most complete 2D scene understanding.

Modern semantic segmentation models are built on specialized neural network architectures:

• Fully Convolutional Networks (FCNs): The foundational architecture, replacing fully-connected layers with convolutional layers to output spatial maps instead of single labels.
• Encoder-Decoder Networks: The encoder (e.g., ResNet, VGG) downsamples the image to extract high-level features. The decoder (e.g., U-Net, SegNet) upsamples these features to the original resolution for pixel-wise prediction.
• Dilated/Atrous Convolutions: Used in models like DeepLab. These convolutions expand the receptive field without reducing spatial resolution, capturing multi-scale context crucial for labeling large objects (e.g., 'building') and small details.
• Attention Mechanisms & Transformers: Architectures like Segmenter and Swin Transformer use self-attention to model long-range dependencies across the entire image, improving label consistency for large, coherent regions.

The principle extends directly to 3D perception, which is critical for robotics and autonomous driving:

• 3D Semantic Segmentation: Classifies every point in a point cloud (from LiDAR or depth cameras) into categories. This is computationally challenging due to data sparsity and irregular structure.
• Key Architectures: PointNet and PointNet++ directly process unordered point sets. Sparse Convolutional Networks operate efficiently on voxelized point clouds. Range View methods project point clouds to 2D for processing with standard CNNs.
• Applications: Understanding 3D scenes for robot navigation, constructing semantic maps, and identifying drivable surfaces or obstacles in autonomous vehicle perception stacks.

Model performance is measured using metrics that account for pixel-wise accuracy and class imbalance:

• Pixel Accuracy: The percentage of correctly classified pixels. Simple but misleading if classes are imbalanced (e.g., a large 'road' region).
• Mean Intersection over Union (mIoU): The standard benchmark. For each class, IoU is the area of overlap between the predicted and ground truth segmentation, divided by the area of union. The mean is taken over all classes. This penalizes both false positives and false negatives.
• Frequency Weighted IoU (FWIoU): A variant of mIoU that weights each class's IoU by its pixel frequency, accounting for class prevalence.
• Per-Class Metrics: Precision, recall, and Dice coefficient (F1 score) are also reported for each semantic category to diagnose specific failures.

Semantic segmentation is a foundational capability for numerous real-world systems:

• Autonomous Driving: Understanding drivable space, lanes, and all actors (Cityscapes, BDD100K, KITTI datasets).
• Medical Image Analysis: Delineating tumors, organs, or cells in MRI/CT scans (BraTS, MSD datasets).
• Robotics & AR/VR: For scene understanding, object manipulation, and environment overlay.
• Satellite & Aerial Imagery: Land use/cover classification, urban planning.
• Industrial Inspection: Identifying defects or components on assembly lines.

Notable Datasets:

• Cityscapes: High-quality urban street scenes.
• ADE20K: A diverse scene parsing dataset with 150 categories.
• PASCAL VOC: A classic benchmark for object segmentation.
• ScanNet: A large-scale dataset of 3D indoor scenes with semantic annotations.

Instance segmentation is the computer vision task that identifies and delineates each distinct object of interest within a scene, assigning a unique label to every pixel or point belonging to each individual instance. It goes beyond semantic segmentation by differentiating between objects of the same class.

• Key Distinction: While semantic segmentation labels all 'car' pixels as 'car', instance segmentation identifies 'car #1', 'car #2', etc.
• Common Architectures: Mask R-CNN is a foundational model for 2D instance segmentation. For 3D point clouds, methods like PointGroup or 3D-BoNet are used.
• Primary Output: A mask or label map where each object instance has a unique identifier.

Panoptic segmentation is a unified computer vision task that combines semantic segmentation (labeling every pixel with a class) and instance segmentation (identifying individual objects) to provide a complete scene understanding.

• Two Segment Types: It labels all pixels as either 'stuff' (amorphous regions like sky, road) or 'things' (countable objects like cars, people).
• 'Things' receive instance IDs; 'stuff' receives only a semantic class label.
• Evaluation Metric: The standard metric is the Panoptic Quality (PQ) score, which balances recognition and segmentation quality.

A point cloud is a discrete set of data points in a three-dimensional coordinate system, typically representing the external surfaces of objects or environments as captured by sensors like LiDAR or depth cameras.

• Primary Data Source: The raw output for 3D semantic segmentation in robotics and autonomous driving.
• Challenges: Data is unordered, irregular, and sparse, requiring specialized neural architectures like PointNet or PointNet++ for direct processing.
• Attributes: Each point often has coordinates (x, y, z) and may include additional features like intensity or RGB color.

3D object detection is the task of identifying and localizing objects within a 3D space, typically by predicting their oriented 3D bounding boxes, class labels, and sometimes orientation from sensor data like point clouds or images.

• Output vs. Segmentation: Provides a bounding box rather than a per-point mask. It answers 'where is the object?' rather than 'what is the exact shape?'
• Common Representations: Input data is often converted into voxel grids or Bird's-Eye View (BEV) maps for processing with CNNs.
• Key Models: SECOND, PointPillars, and PV-RCNN are standard architectures for LiDAR-based 3D detection.

Bird's-Eye View (BEV) is a top-down, orthographic projection representation of a scene, commonly used in autonomous driving to unify perceptions from multiple sensors into a common 2D or 2.5D spatial plane for planning.

• Role in Segmentation: Modern vision-centric autonomous systems often perform BEV semantic segmentation, projecting image features from multiple cameras into a unified BEV space for scene understanding.
• Advantage: Provides a metric space where spatial relationships are consistent, simplifying tasks like path planning.
• Techniques: Uses transformer-based 'lift-splat-shoot' or cross-attention mechanisms to create the BEV feature map.

An occupancy grid is a probabilistic, discrete representation of a 3D environment where each cell (voxel) stores the probability that it is occupied by an obstacle, commonly used in robotics for mapping and navigation.

• Relation to Segmentation: 3D semantic segmentation can be used to create a semantic occupancy grid, where each voxel is labeled with a class (e.g., free, car, vegetation) instead of just a binary occupancy probability.
• Use Case: Critical for motion planning, as it provides a voxelized 3D map of traversable and non-traversable space.
• Challenge: Balancing resolution with memory and computational cost.