Semantic Segmentation

The Fully Convolutional Network (FCN) is the foundational architecture that replaced fully connected layers with convolutional layers, enabling dense pixel-wise prediction from an input image of any size. Key innovations include:

• Encoder-Decoder Structure: The encoder downsamples the image to extract high-level features, while the decoder upsamples to restore spatial resolution for pixel-level labels.
• Skip Connections: These fuse high-resolution, low-level feature maps from the encoder with the upsampled, semantically rich maps in the decoder, preserving fine-grained spatial details crucial for accurate boundaries.
• This architecture established the template for nearly all modern semantic segmentation models.

U-Net is a seminal encoder-decoder architecture specifically designed for biomedical image segmentation but widely adopted across domains. Its defining characteristics are:

• Symmetrical U-Shape: Features a contracting path (encoder) to capture context and an expansive path (decoder) for precise localization.
• Extensive Skip Connections: Copies and concatenates feature maps from each encoder stage to the corresponding decoder stage, providing the decoder with both high-level semantics and low-level spatial information.
• Precise Localization: This design makes U-Net exceptionally effective even with limited training data, as it leverages all available feature hierarchies.

Dilated (or Atrous) Convolutions are a critical technique for increasing the receptive field of convolutional layers without increasing the number of parameters or losing resolution through pooling.

• Mechanism: They insert spaces (holes) between the kernel elements, allowing the filter to sample a wider area of the input.
• Purpose: This enables the network to capture multi-scale contextual information at a single layer, which is vital for understanding large objects (e.g., 'sky', 'road') while maintaining a dense feature map for precise segmentation.
• Architectures like DeepLab series heavily utilize atrous convolutions and atrous spatial pyramid pooling (ASPP) to segment objects at multiple scales effectively.

Vision Transformers (ViTs) and their derivatives have been adapted for segmentation by modeling long-range dependencies across the entire image.

• Vision Transformer (ViT): Splits an image into patches, projects them into tokens, and processes them with a standard Transformer encoder, capturing global context.
• Segmentation Adaptations: Models like Segmenter and SegFormer use a Transformer encoder to create a global feature representation. A lightweight decoder then upsamples these features into a pixel-wise segmentation map.
• Swin Transformer: Uses a hierarchical shifted-window mechanism to compute self-attention locally within non-overlapping windows, making it computationally efficient for dense prediction tasks while still capturing cross-window connections.

For on-device 3D reconstruction and mobile AR, segmentation models must be fast and lightweight. Key architectures and techniques include:

• MobileNet & EfficientNet Backbones: Using these efficient feature extractors as the encoder reduces computational cost.
• Lightweight Decoders: Architectures like Fast-SCNN or BiSeNet use a two-path network—one for spatial detail and one for context—that are fused, achieving real-time speeds.
• Model Compression: Techniques like pruning and quantization (e.g., INT8) are applied post-training to these architectures to further reduce memory footprint and latency for embedded vision systems.

Instance Segmentation extends semantic segmentation by not only classifying each pixel but also distinguishing between different objects of the same class.

• Key Difference: Semantic segmentation labels all 'car' pixels the same. Instance segmentation gives a unique label to each individual car.
• Architectural Approaches:
  • Two-Stage (Mask R-CNN): First detects object bounding boxes, then segments the object within each box.
  • Single-Stage (YOLACT, SOLO): Predicts masks in a single forward pass, trading some accuracy for significant speed gains, which is valuable for real-time applications.
• This task is crucial for applications requiring object-level interaction, such as robotic manipulation.

A more granular computer vision task that not only classifies each pixel by object class (semantic segmentation) but also distinguishes between individual object instances. For example, it would label all pixels for 'car' and also differentiate Car A from Car B from Car C.

• Key Distinction: Semantic segmentation labels 'stuff' (sky, road) and 'things' (cars) by category. Instance segmentation only labels countable 'things' and gives each a unique ID.
• Common Architectures: Extends semantic segmentation models like Mask R-CNN, which adds a parallel branch for predicting object masks and uses bounding box proposals for instance separation.

A unified vision task that combines semantic segmentation (for amorphous 'stuff' like sky and road) and instance segmentation (for countable 'things' like people and cars) into a single, comprehensive output. Every pixel in the image receives a unique label pair: a semantic class and, for 'things', an instance ID.

• Goal: Provide a complete scene parsing. The output is often visualized with a unique color for each thing instance and consistent colors for each stuff region.
• Evaluation: Uses the Panoptic Quality (PQ) metric, which balances recognition quality (segmentation accuracy) and detection quality (instance identification).

The task of predicting the distance (depth) from the camera to each point in the scene. While semantic segmentation provides a 2D label map, depth estimation provides a 2.5D geometry map. Combining the two is powerful for 3D scene understanding.

• Methods: Can be monocular (from a single image), stereo (using two cameras), or active (using sensors like LiDAR or ToF cameras).
• Integration with Segmentation: Depth-aware semantic segmentation networks use depth cues as an additional input channel to improve boundary accuracy and contextual understanding, especially for objects at similar scales.

A foundational algorithm for robotics and AR that allows a device to build a map of an unknown environment while simultaneously tracking its own location within it. Semantic segmentation directly enhances SLAM systems.

• Semantic SLAM: Integrates pixel-level class labels into the mapping process. Instead of creating a map of generic geometric features, it creates a map of semantically labeled objects (e.g., walls, chairs, doors).
• Benefits: Enables higher-level reasoning (e.g., 'navigate to the kitchen'), improves loop closure detection by recognizing objects, and makes maps more human-interpretable and useful for long-term autonomy.

A volumetric representation used in real-time 3D reconstruction (like in KinectFusion) to fuse multiple depth frames into a consistent 3D model. Semantic segmentation can be integrated to create semantic TSDFs.

• How it works: Each voxel in a 3D grid stores the signed distance to the nearest surface (negative inside, positive outside), truncated to a limit. As new depth frames are integrated, the values are averaged.
• Semantic Fusion: When a depth frame has an associated semantic label map (from a segmented RGB image), the label is projected into 3D and fused into the TSDF volume. This results in a dense 3D model where every voxel has both a geometry value and a semantic class label.

The execution of a trained machine learning model directly on an end-user device (e.g., smartphone, robot, IoT sensor) without a cloud connection. Running semantic segmentation on-device is critical for applications requiring low latency, privacy, and offline operation.

• Challenges: Semantic segmentation models are computationally intensive due to their high-resolution, pixel-wise outputs. This demands significant optimization for edge deployment.
• Optimization Techniques: To enable real-time on-device segmentation, engineers use:
  • Model Compression: Techniques like pruning and quantization (e.g., INT8) to reduce model size and accelerate computation.
  • Efficient Architectures: Designing networks with fewer parameters, like MobileNetV3 or EfficientNet backbones combined with lightweight decoder heads.
  • Hardware Acceleration: Leveraging dedicated Neural Processing Units (NPUs) or GPU shaders on mobile SoCs for optimal performance and power efficiency.