Semantic Segmentation

Unlike object detection which outputs bounding boxes, semantic segmentation produces a classification map with the same spatial resolution as the input image. Each pixel is independently assigned to a predefined category such as 'liver', 'tumor', or 'background'. This dense prediction paradigm enables precise boundary delineation and volumetric quantification of anatomical structures, making it foundational for radiological analysis where exact organ contours matter more than approximate locations.

A defining constraint of semantic segmentation is that all objects of the same class are merged into a single, undifferentiated region. Two adjacent tumor nodules receive the identical 'tumor' label with no separation boundary. This contrasts with instance segmentation, which assigns unique identifiers to each distinct object. For medical applications, this property is advantageous when measuring total organ volume or aggregate disease burden, but insufficient when counting or tracking individual lesions.

Modern semantic segmentation networks rely on an encoder-decoder structure with skip connections. The encoder—typically a ResNet or Vision Transformer—progressively downsamples the image to extract hierarchical, high-level semantic features. The decoder upsamples these features back to the original resolution, using skip connections to recover fine spatial details lost during encoding. Architectures like U-Net and DeepLabV3+ exemplify this design, achieving state-of-the-art accuracy in medical imaging tasks.

Accurate segmentation requires understanding both local texture and global anatomical context. Techniques like Atrous Spatial Pyramid Pooling (ASPP) apply dilated convolutions at multiple rates in parallel, capturing features at different receptive field sizes without losing resolution. This allows the network to simultaneously recognize a pixel as 'liver tissue' based on local texture while using surrounding organ context to disambiguate it from similar-appearing adjacent structures like the spleen or kidney.

Medical segmentation datasets exhibit extreme class imbalance—background pixels vastly outnumber foreground organ or lesion pixels. Standard cross-entropy loss fails in this regime. Specialized loss functions address this:

• Dice Loss: Optimizes the spatial overlap coefficient directly, robust to imbalance
• Tversky Loss: Adds weights to penalize false negatives more heavily than false positives, critical when missing a tumor is clinically unacceptable
• Focal Loss: Down-weights easy background examples, forcing the network to focus on hard boundary pixels

Raw network outputs often produce noisy, irregular boundaries. Conditional Random Fields (CRFs) refine predictions by modeling the pairwise agreement between neighboring pixels based on both spatial proximity and intensity similarity. A pixel is more likely to share a label with a nearby pixel of similar intensity. While modern architectures increasingly integrate boundary refinement end-to-end, CRFs remain valuable as a post-hoc correction step, particularly for fine structures like vessel walls or organ capsules where pixel-level precision is paramount.

The foundational encoder-decoder convolutional network for biomedical segmentation. Its symmetric structure compresses spatial context via a contracting path, then recovers precise localization via an expanding path. Skip connections directly transfer high-resolution features from encoder to decoder layers, preserving fine boundary details lost during downsampling.

The primary evaluation metric for segmentation accuracy, also known as the Jaccard Index. IoU quantifies the ratio of the overlapping area to the total union area between a predicted mask and ground truth.

• Formula: TP / (TP + FP + FN)
• Range: 0.0 (no overlap) to 1.0 (perfect match)
• Threshold: An IoU > 0.5 is typically considered a correct detection in instance segmentation benchmarks.

Semantic segmentation assigns a class to every pixel but cannot distinguish between separate objects of the same class. Two related tasks solve this:

• Instance Segmentation: Detects and masks each distinct object (e.g., individual cells). Combines object detection with pixel-level masks.
• Panoptic Segmentation: Unifies both tasks, assigning a class and instance ID to countable foreground objects while labeling amorphous background regions (sky, road) semantically.

A statistical measure of spatial overlap, functionally equivalent to the F1 Score applied to pixel-level classification. It is the most common loss function for training medical segmentation models.

• Formula: 2 * |A ∩ B| / (|A| + |B|)
• Advantage: Handles extreme class imbalance better than pixel-wise cross-entropy, as it focuses on the overlap of the often-small foreground region.

A probabilistic graphical model used as a post-processing step to refine coarse segmentation outputs. CRFs model each pixel as a node and encourage label agreement between neighboring pixels with similar intensities.

• Fully Connected CRFs enforce long-range smoothness.
• Limitation: Largely superseded by end-to-end deep learning architectures that learn boundary constraints directly.

A critical clinical application of semantic segmentation in radiotherapy planning. OARs are healthy anatomical structures surrounding a tumor that are sensitive to radiation (e.g., spinal cord, parotid glands).

• Goal: Precisely delineate OARs to calculate radiation dose constraints and minimize collateral toxicity.
• Standard: Segmentation results are stored in DICOM RT Structure Sets for treatment planning systems.