Group LASSO

Unlike standard LASSO which selects individual features, Group LASSO applies an L2,1 mixed norm penalty. The objective function minimizes the sum of squared errors plus a penalty term proportional to the L2 norm of each group's coefficient vector. This forces the entire coefficient vector for a group to zero simultaneously, achieving structured sparsity that respects natural feature groupings such as gene pathways or transcription factor targets.

The optimization problem is: minimize (1/2)||y - Σ X_g β_g||²₂ + λ Σ √p_g ||β_g||₂, where:

• X_g is the feature matrix for group g
• β_g is the coefficient vector for group g
• p_g is the group size
• λ controls overall regularization strength The √p_g term accounts for varying group sizes, preventing larger groups from being unfairly penalized. This is a convex optimization problem solvable via block coordinate descent.

The standard formulation assumes non-overlapping groups, but a powerful extension allows features to belong to multiple groups simultaneously. This is critical in genomics where a gene may participate in several biological pathways. The overlapping variant uses a latent variable approach, duplicating features that appear in multiple groups and applying the group penalty to each copy independently, then constraining the copies to sum to the original coefficient value.

This hybrid combines group-level and within-group sparsity by adding both an L1 penalty on individual coefficients and an L2,1 penalty on groups: λ₁||β||₁ + λ₂ Σ ||β_g||₂. This is particularly valuable when you want to select entire pathways but also allow only the most predictive genes within a selected pathway to have non-zero weights, yielding a doubly sparse solution that enhances interpretability in biomarker panels.

The standard optimization algorithm iterates over groups, holding all other groups fixed while solving a subproblem for each group. For group g, the update rule has a closed form: β_g = (1 - λ√p_g / ||S_g||₂)₊ S_g, where S_g is the partial residual solution and (·)₊ denotes the positive part. This is a soft-thresholding operator applied to the group norm, efficiently shrinking entire groups to zero when their collective contribution falls below the threshold.

Group LASSO excels in SNP-to-gene mapping where all SNPs within a gene form a natural group, and in pathway-based biomarker selection where gene sets from KEGG or Gene Ontology define groups. It also handles categorical variable selection in clinical models—treating all dummy-encoded levels of a categorical predictor as one group ensures the entire variable is selected or discarded, preventing the interpretability problem of partial categorical inclusion.

Group LASSO replaces the sum of absolute coefficients with a sum of L2 norms for each feature group. The objective function minimizes: RSS + λ * Σ √(p_g) * ||β_g||₂, where p_g is the group size and β_g is the coefficient vector for group g. This non-differentiable penalty at the origin forces the entire group's coefficients to zero simultaneously, achieving block-wise sparsity. The √(p_g) factor compensates for varying group sizes, preventing larger groups from being unfairly penalized.

Groups are defined using prior biological knowledge to ensure interpretable selection:

• Gene pathways: KEGG, Reactome, or Gene Ontology sets group genes by shared function
• SNP sets: Variants grouped by gene, regulatory region, or linkage disequilibrium block
• Protein domains: InterPro or Pfam families grouping structurally related regions
• Metabolite classes: Lipid subclasses or amino acid derivatives
• Co-expression modules: WGCNA-derived clusters of correlated transcripts This domain-informed grouping prevents the selection of isolated, uninterpretable features.

The optimization is typically solved via block coordinate descent or proximal gradient methods:

• Block coordinate descent: Iteratively updates each group's coefficients while holding others fixed, applying a group-wise soft-thresholding operator: β_g = (1 - λ√(p_g) / ||S_g||₂)₊ * S_g, where S_g is the unpenalized update
• Proximal gradient descent: Applies the group soft-thresholding as a proximal operator after each gradient step
• SGL package in R and scikit-learn's group_lasso extensions provide production-ready implementations
• Cross-validation selects the optimal λ by minimizing prediction error across held-out folds

The Sparse Group LASSO combines group-level and within-group sparsity by adding both penalties: λ₁ * Σ ||β_g||₂ + λ₂ * ||β||₁. This allows selection of entire groups while also zeroing out individual features within selected groups. In biomarker studies, this is invaluable when a pathway is relevant but only a subset of its member genes are truly predictive. The dual penalty requires tuning two hyperparameters, typically via a grid search over (λ₁, λ₂) pairs.

Real biological groups often overlap—a gene may belong to multiple pathways. Extensions handle this complexity:

• Overlap Group LASSO: Uses a latent variable decomposition where each feature's coefficient is the sum of contributions from all groups it belongs to, penalizing each group's contribution separately
• Hierarchical Group LASSO: Enforces that a child group (e.g., a specific protein complex) can only be selected if its parent group (e.g., the broader pathway) is also selected, reflecting biological hierarchies These methods prevent double-penalization of shared features and maintain ontological consistency.

A typical Group LASSO pipeline for biomarker identification:

1. Preprocessing: Normalize expression data, handle batch effects with ComBat, standardize features to zero mean and unit variance
2. Group definition: Map probes/genes to curated pathway databases or define custom gene sets from prior experiments
3. Model fitting: Run Group LASSO with 10-fold cross-validation to select λ that minimizes mean squared error
4. Stability assessment: Apply the model to 100 bootstrap resamples; retain groups selected in >80% of iterations
5. Validation: Test the selected pathway biomarkers in an independent cohort for prognostic or predictive performance

The foundational method that Group LASSO extends. LASSO adds a penalty equal to the absolute value of coefficient magnitudes to the loss function, shrinking irrelevant feature coefficients to exactly zero. Unlike Group LASSO, it performs individual feature selection without considering group structure. The penalty term is λ∑|βⱼ|, creating a diamond-shaped constraint region that forces sparsity at the vertex intersections.

A direct extension that combines both group-level and within-group sparsity. The penalty includes a standard Group LASSO term plus an additional L1 penalty on individual coefficients: λ₁∑||β_g||₂ + λ₂||β||₁. This allows the model to select entire groups while also zeroing out irrelevant features within selected groups. Particularly useful in pathway analysis where entire biological pathways may be relevant, but only specific genes within them drive the signal.

A variant that handles features belonging to multiple overlapping groups simultaneously. The penalty uses a latent variable formulation where each feature's coefficient is decomposed into components corresponding to each group it belongs to. The penalty becomes λ∑||β̄_g||₂, where β̄_g are the coefficient subvectors for each group. Critical for gene set analysis where genes participate in multiple biological pathways.

Combines L1 and L2 penalties linearly: λ₁||β||₁ + λ₂||β||₂². While not group-structured, Elastic Net handles highly correlated features by selecting or deselecting them together—a behavior Group LASSO achieves through explicit group definition. The L2 component stabilizes the L1 path when features are correlated, avoiding the arbitrary single-variable selection that standard LASSO exhibits among correlated predictors.

A resampling-based framework that can wrap around Group LASSO to improve selection robustness. It applies the selection algorithm to many random subsamples of the data and selects features or groups that are consistently chosen above a threshold frequency. Addresses Group LASSO's sensitivity to the regularization parameter λ by integrating over a range of penalty values, providing finite-sample false discovery control without asymptotic assumptions.

Enforces tree-structured sparsity where a feature can only be selected if its ancestors in a predefined hierarchy are also selected. The penalty is a sum of group norms over nested groups defined by the tree structure. Applied in taxonomic microbiome analysis and wavelet decomposition where features naturally organize into parent-child relationships. Contrasts with Group LASSO's flat, non-hierarchical group structure.