Segment Anything Model (SAM)

SAM's core innovation is its ability to generate valid segmentation masks from diverse, ambiguous prompts without retraining. It accepts sparse prompts (points, boxes) and dense prompts (masks) simultaneously.

• Point prompts: A single foreground/background click guides segmentation.
• Box prompts: A bounding box loosely enclosing the target object.
• Mask prompts: A coarse, low-resolution mask is refined into a precise output.
• Ambiguity awareness: When a prompt is ambiguous (e.g., a point on a shirt collar could mean the collar, shirt, or person), SAM outputs multiple valid masks with confidence scores.

SAM uses a heavy-once, light-many design pattern. A massive ViT-based image encoder runs once per image to generate an embedding. A lightweight prompt encoder and mask decoder then run in real-time for each user interaction.

• Image encoder: A pre-trained Vision Transformer (ViT-H, ViT-L, ViT-B) creates a dense image embedding.
• Prompt encoder: Encodes geometric prompts (points, boxes) using positional encodings and text prompts via CLIP.
• Mask decoder: A lightweight transformer that cross-attends prompt tokens to image embeddings, predicting masks and IoU scores in ~50ms on GPU.

SAM is trained to perform zero-shot transfer—segmenting objects it has never seen during training without fine-tuning. This is achieved through a unique task design and dataset scale.

• Task formulation: The model is trained on a promptable segmentation task, not specific object classes.
• SA-1B dataset: Trained on 11 million images with over 1 billion masks, the largest segmentation dataset ever created.
• Emergent behavior: SAM can segment corners of rooms, reflections in mirrors, and occluded objects—scenarios not explicitly labeled in training data.
• Domain gap: While general-purpose, SAM may require fine-tuning (e.g., MedSAM) for specialized domains like medical imaging where anatomical boundaries differ from natural images.

SAM was trained using a three-stage data engine that iteratively improved both the model and the dataset. The resulting SA-1B dataset is a public good for computer vision research.

• Stage 1 - Assisted manual: Human annotators clicked on objects; SAM suggested masks. This generated 4.3M masks on 120k images.
• Stage 2 - Semi-automatic: SAM automatically proposed masks on 180k images; annotators refined and verified them, yielding 5.9M masks.
• Stage 3 - Fully automatic: SAM ran in automatic mode on 11M images, generating 1.1B masks with confidence scores.
• Scale: SA-1B contains 400x more masks than any existing segmentation dataset.

While SAM excels on natural images, its zero-shot performance degrades on medical modalities. MedSAM addresses this by fine-tuning SAM on a large-scale medical dataset.

• Training data: 1.5 million image-mask pairs across 10 modalities (CT, MRI, X-ray, ultrasound, endoscopy, etc.) and 30+ cancer types.
• Architecture: Uses SAM's ViT-B encoder with adapter layers, fine-tuned end-to-end.
• Prompting: Supports bounding box prompts, which are standard outputs from detection models in clinical workflows.
• Performance: Achieves state-of-the-art Dice scores on multi-organ CT segmentation and outperforms specialist models like nnU-Net on certain tasks.

A domain-specific fine-tuning of SAM on 1.5 million medical image-mask pairs spanning 10 modalities and 30+ anatomical structures. MedSAM addresses SAM's limitations on medical data—where boundaries are subtle, contrast is low, and objects lack clear visual separation. The adaptation uses bounding box prompts as the primary input modality, achieving state-of-the-art performance on diverse segmentation tasks from CT organ delineation to ultrasound lesion detection.