Cell Segmentation

The primary metric for segmentation quality, measuring how precisely predicted cell boundaries match ground-truth annotations. Mean Average Precision (mAP) at various Intersection-over-Union (IoU) thresholds is the standard benchmark. State-of-the-art models achieve mAP > 0.85 on curated datasets like CellPose and TissueNet.

• IoU (Jaccard Index): Ratio of overlap area to union area between predicted and ground-truth masks
• Dice Coefficient (F1): Harmonic mean of precision and recall for pixel-level classification
• Boundary F-score: Specifically evaluates edge alignment rather than region overlap

The ability to distinguish touching or overlapping cells as distinct objects rather than merging them into a single mask. This is critical for densely packed tissues like tumors or lymphoid organs.

• Watershed-based post-processing: Uses distance transforms to split merged nuclei
• Flow-field prediction: Models like CellPose predict spatial gradients pointing toward cell centers
• Star-convex polygon fitting: Approximates cell shapes as star-convex polygons for efficient separation
• Graph-cut refinement: Formulates separation as an energy minimization problem on pixel adjacency graphs

The capacity to segment cells with highly variable shapes and sizes without retraining. A robust algorithm handles elongated neurons, irregular epithelial cells, and spherical lymphocytes within the same tissue section.

• Scale invariance: Detects cells ranging from 5μm lymphocytes to 100μm neurons
• Shape prior learning: Learns implicit shape distributions rather than assuming circularity
• Domain adaptation: Transfers performance from training cell types to novel morphologies
• Failure mode: Classical thresholding methods often fail on non-spherical or elongated cells

Processing speed measured in megapixels per second or cells per minute, determining feasibility for whole-slide images (WSIs) that can exceed 10 gigapixels. GPU-accelerated inference is essential for production pipelines.

• Real-time segmentation: < 100ms per 1024×1024 tile for interactive applications
• Batch processing: Parallel tile processing with overlap-aware stitching
• Memory footprint: Peak GPU memory usage during inference, critical for edge deployment
• Trade-off: Higher accuracy models (e.g., transformer-based) often require 2-5× more compute than CNN-based alternatives

The ability to leverage multiple imaging channels simultaneously—such as DAPI nuclear stain, membrane markers, and cytoplasmic signals—to improve segmentation accuracy beyond single-channel methods.

• Nuclear-guided segmentation: Uses DAPI/Hoechst channel to seed cell locations, then expands to membrane boundaries
• Attention-based fusion: Learns channel importance weights dynamically per spatial location
• Missing modality robustness: Gracefully degrades when expected channels are absent
• Application: Essential for multiplexed imaging technologies like CODEX, MIBI, and CyCIF

The estimation of confidence scores for each predicted mask, enabling downstream quality control and human-in-the-loop review. Critical for clinical and regulatory applications where segmentation errors propagate to biological conclusions.

• Epistemic uncertainty: Model uncertainty due to limited training data, reducible with more samples
• Aleatoric uncertainty: Inherent noise from ambiguous boundaries or imaging artifacts
• Monte Carlo dropout: Practical approximation using multiple stochastic forward passes
• Output: Per-pixel or per-cell confidence maps flagging regions for manual review

The primary application of cell segmentation in spatial transcriptomics is the precise identification of individual cell membranes or nuclei to assign each detected mRNA molecule to its cell of origin. Modern deep learning models like Cellpose and StarDist use convolutional neural networks to predict spatial gradients and star-convex polygons, respectively, enabling robust segmentation even in densely packed tissues.

• Membrane staining (e.g., anti-CD45) provides direct boundaries but requires specialized chemistry.
• Nucleus-only segmentation (DAPI) is simpler but can miss cytoplasmic transcripts.
• Label-free methods use transmitted light or autofluorescence to predict boundaries without staining artifacts.

Once cell boundaries are defined, the segmentation mask serves as a spatial index to assign each detected RNA molecule to a specific cell ID. This process, often called molecular assignment or transcript decoding, is critical for building the single-cell expression matrix used in all downstream analyses.

• Point-in-polygon algorithms test whether each transcript coordinate falls within a segmented cell boundary.
• Probabilistic assignment models account for segmentation uncertainty by assigning transcripts to cells with a confidence score.
• Expansion microscopy physically enlarges tissues to improve assignment accuracy in subcellular resolution assays like MERFISH.

Segmented cells provide the unit of observation for cell-type annotation—the process of labeling each cell based on its expression profile. After segmentation, clustering algorithms (e.g., Leiden, Louvain) group cells by transcriptional similarity, and marker gene databases (e.g., CellMarker, PanglaoDB) are used to assign biological identities.

• Automated annotation tools like CellTypist and scType use logistic regression or neural networks trained on reference atlases.
• Spatial context can refine annotations by incorporating neighborhood composition into the classification.
• Hierarchical annotation resolves cell types at multiple granularities, from broad lineages to rare subtypes.

Beyond transcript counts, segmentation masks enable the extraction of morphological features that provide orthogonal information about cell state. These features—including cell area, perimeter, eccentricity, and solidity—can distinguish activated immune cells, senescent cells, or malignant morphologies.

• Nuclear texture features (Haralick, Gabor) quantify chromatin organization patterns.
• Shape descriptors like Zernike moments capture complex boundary irregularities.
• Integrated spatial-morphological analysis combines these features with gene expression for multimodal cell state characterization.

Accurate segmentation is the prerequisite for reconstructing tissue architecture—the spatial organization of cell types into functional units. By combining cell boundaries with cell-type labels, researchers can identify cellular neighborhoods, tissue domains, and spatial gradients that define organ function.

• Spatial neighborhood graphs connect each cell to its k-nearest neighbors for graph-based domain detection.
• Spatial autocorrelation metrics like Moran's I quantify the degree of cell-type clustering.
• Niche analysis identifies recurrent multicellular motifs associated with disease progression or treatment response.

Segmentation outputs serve as a critical quality control checkpoint in spatial transcriptomics pipelines. Cells with aberrant morphology or transcript counts are flagged and filtered before downstream analysis to prevent artifacts from propagating.

• Doublet detection identifies segmentation masks containing two or more cells based on total transcript counts or mixed expression profiles.
• Damaged cell filtering removes cells with high mitochondrial gene expression or fragmented morphology.
• Segmentation confidence scores from models like Cellpose allow users to set thresholds for cell inclusion, balancing sensitivity and specificity.

The computational process of partitioning a digital tissue image into distinct anatomical or functional regions based on pixel-level classification. Unlike cell segmentation, which delineates individual cells, tissue segmentation identifies larger structures such as tumor stroma, necrotic cores, or cortical layers. Deep learning architectures like U-Net and DeepLab are commonly employed, often leveraging convolutional neural networks trained on pathologist-annotated whole-slide images to generate tissue-level masks that guide region-specific downstream analyses.

A computational process that estimates the proportions of different cell types within a spatial transcriptomics spot by separating the mixed gene expression signal. When cell segmentation is impossible due to resolution limits, deconvolution infers cellular composition from bulk-like expression profiles using reference single-cell RNA-seq signatures. Algorithms like RCTD, SPOTlight, and cell2location employ regression-based or probabilistic models to map cell-type abundances back onto tissue coordinates, bridging the gap between low-resolution spatial data and single-cell biology.

A data structure where each segmented cell or spatial location is a node, and edges connect neighboring locations based on a distance threshold or k-nearest neighbors. This graph representation is the computational backbone for spatial-aware machine learning. Once cells are segmented, their centroids form the nodes, enabling graph neural networks to learn context-aware representations, spatial autocorrelation statistics to quantify patterning, and ligand-receptor co-localization analyses to identify communicating cell pairs within the tissue microenvironment.

The unsupervised identification of tissue regions with coherent gene expression profiles and histology, often achieved through graph-based clustering or hidden Markov models. After cell segmentation, each cell's expression vector and spatial coordinates are used to group cells into domains that reflect functional anatomy. Methods like BayesSpace and SpaGCN leverage spatial neighborhood graphs and deep learning to enhance clustering resolution, enabling the discovery of spatially variable genes and cellular niches that define tissue architecture.

A statistical measure of the degree to which a variable's values at nearby locations in a tissue are more similar than expected by random chance. After cell segmentation, Moran's I and Geary's C quantify whether a gene's expression is randomly distributed, clustered, or dispersed across the segmented cell population. Positive spatial autocorrelation indicates that neighboring cells share similar expression states, often reflecting cell-cell communication, clonal expansion, or shared microenvironmental gradients that drive tissue organization.

The computational alignment of multiple tissue images or spatial datasets into a common coordinate system to enable integrative cross-modality analysis. Cell segmentation masks from H&E-stained images are registered to spatial transcriptomics coordinates, allowing gene expression to be mapped onto histological features. Non-linear image registration algorithms like Elastix and ANTs correct for tissue deformation, enabling the fusion of segmented cell boundaries with multiplexed immunofluorescence or in situ sequencing data for multi-omic single-cell atlases.