Hierarchical Clustering

The most common strategy in patient stratification. The algorithm initializes by treating every patient as an individual cluster. It then iteratively merges the two most similar clusters based on a chosen linkage criterion until a single root cluster remains. This process creates a nested hierarchy where early merges represent highly similar patients, and later merges capture broader disease categories. The sequence of merges is irreversible, meaning once patients are joined, they cannot be separated in subsequent steps.

A less common but computationally intensive alternative. The algorithm starts with all patients in a single cluster and recursively splits them into smaller, more homogeneous groups. At each step, a flat clustering algorithm like k-means is applied to bisect the selected cluster. This method is particularly useful when the primary interest lies in identifying the dominant, high-level splits in a heterogeneous disease cohort, such as separating inflammatory from non-inflammatory subtypes before further refinement.

The mathematical rule defining the distance between two clusters, critically influencing dendrogram topology:

• Single Linkage: Minimum distance between any two points in different clusters. Prone to chaining effects.
• Complete Linkage: Maximum distance. Tends to produce compact, equally sized clusters.
• Average Linkage: Mean distance between all pairs. A balanced compromise for most omics data.
• Ward's Method: Minimizes the increase in total within-cluster variance after merging. Often preferred for genomic data as it produces clusters of similar size and minimizes information loss.

A dendrogram is a tree diagram encoding the entire clustering history. The vertical axis represents the distance or dissimilarity at which merges occur. Clinicians can 'cut' the tree at a specific height to obtain a desired number of patient subgroups. Long, vertical branches indicate well-separated, stable clusters, while short branches suggest less distinct groupings. Cophenetic correlation coefficient quantifies how faithfully the dendrogram preserves the original pairwise distances, serving as a quality metric.

The choice of dissimilarity measure is fundamental to biological interpretation:

• Euclidean Distance: Standard for continuous clinical variables like lab values.
• Manhattan Distance: Robust to outliers, useful for high-dimensional omics data.
• Cosine Similarity: Focuses on the angular relationship between patient profiles, ignoring magnitude differences—ideal for RNA-seq count data where sequencing depth varies.
• Jaccard Distance: Used for binary presence/absence data, such as mutation profiles or symptom checklists.

Standard agglomerative clustering has a time complexity of O(n³) and memory complexity of O(n²), where n is the number of patients. This makes it prohibitive for large biobank-scale cohorts without optimization. Strategies to scale include:

• Pre-clustering with k-means to reduce n before hierarchical analysis.
• Using approximate nearest neighbor graphs.
• Implementing the Lance-Williams algorithm for efficient linkage updates without recalculating the full distance matrix at each step.

The dendrogram is the primary visual output of hierarchical clustering, representing the nested grouping of patients and the order in which clusters are merged. The vertical axis represents the dissimilarity or distance at which merges occur. Key concepts include:

• Cutting the tree: A horizontal slice across the dendrogram defines the final cluster assignments. The height of the cut determines the number of clusters.
• Cophenetic correlation: A metric quantifying how faithfully the dendrogram preserves the original pairwise distances between patients.
• Branch lengths: In evolutionary biology, these represent genetic change; in patient stratification, they reflect the magnitude of molecular difference between subgroups.

Two opposing strategies exist for building the hierarchy:

• Agglomerative (bottom-up): The most common approach. Each patient starts as its own cluster, and pairs of clusters are iteratively merged based on a linkage criterion until a single root cluster remains.
• Divisive (top-down): A less common, computationally intensive approach. All patients start in one cluster, which is recursively split into smaller groups. DIANA (DIvisive ANAlysis) is a classic algorithm. Agglomerative methods are generally preferred for patient stratification due to their computational efficiency and the intuitive nature of the merging process.

The linkage criterion defines how the distance between two clusters of patients is calculated, dramatically impacting the final dendrogram shape:

• Single linkage: Distance between the two closest members of each cluster. Tends to produce long, chain-like clusters and is sensitive to noise.
• Complete linkage: Distance between the two farthest members. Tends to produce compact, equally sized clusters.
• Average linkage (UPGMA): The mean distance between all pairs of patients in the two clusters. A balanced compromise widely used in genomics.
• Ward's method: Minimizes the total within-cluster variance after merging. Tends to produce clusters of similar size and is the default in many bioinformatics packages.

The choice of distance metric is a critical hyperparameter that defines patient similarity. Common choices for molecular data include:

• Euclidean distance: The straight-line distance in feature space. Sensitive to scale; requires careful normalization of gene expression or protein abundance values.
• Manhattan distance: The sum of absolute differences. More robust to outliers in high-dimensional omics datasets.
• Cosine similarity: Measures the angle between patient vectors, ignoring magnitude. Ideal for comparing global gene expression patterns regardless of overall expression level.
• Spearman correlation distance: Based on the rank correlation, capturing non-linear monotonic relationships between patient profiles.

A resampling-based methodology that addresses the instability of a single hierarchical clustering run. The process involves:

• Subsampling the original patient dataset or perturbing the feature space across many iterations.
• Running hierarchical clustering on each subsample.
• Building a consensus matrix where each entry records the proportion of runs in which two patients were assigned to the same cluster. This matrix is then clustered to identify robust, reproducible patient subgroups. This technique is a gold standard in cancer subtyping and is implemented in packages like ConsensusClusterPlus.

Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) extends DBSCAN by constructing a hierarchy of clusters based on density estimates. Unlike traditional hierarchical clustering, HDBSCAN:

• Does not require specifying the number of clusters.
• Robustly identifies clusters of varying density, a common challenge in patient data where rare subtypes exist.
• Explicitly identifies noise points—patients who do not belong to any stable cluster.
• Extracts a flat clustering from the hierarchy based on cluster stability, making it highly practical for identifying both common and rare disease endotypes.