LASSO (L1 Regularization)

LASSO adds a penalty term equal to λ * Σ|βᵢ| (the sum of absolute coefficient values) to the ordinary least squares loss function. This L1 norm geometry creates a constraint region with sharp corners in parameter space. When the elliptical contours of the residual sum of squares hit these corners, coefficients are driven to exactly zero rather than merely approaching zero asymptotically. This is the fundamental distinction from ridge regression (L2), which only shrinks coefficients toward zero but never eliminates them. The sparsity property makes LASSO inherently a feature selection method, automatically discarding irrelevant predictors.

The regularization hyperparameter λ (lambda) controls the strength of the penalty and directly governs model complexity:

• λ = 0: Reduces to ordinary least squares; all features retained
• λ → ∞: All coefficients shrink to zero; intercept-only model
• Intermediate λ: A subset of coefficients become non-zero, with the number selected decreasing as λ increases

Cross-validation is the standard method for selecting an optimal λ, typically using one-standard-error rule to choose the most regularized model within one standard error of the minimum cross-validation error. This yields a more parsimonious model with essentially equivalent predictive performance.

LASSO intentionally introduces bias into coefficient estimates to achieve dramatic reductions in variance. The L1 penalty shrinks coefficient magnitudes away from their unbiased OLS values, but this shrinkage substantially reduces the variance of predictions on new data. The net effect, measured by mean squared error, often favors the biased LASSO estimator over unbiased OLS, particularly when:

• The number of predictors p is large relative to sample size n
• Many true coefficients are zero or near-zero (sparse truth)
• Multicollinearity inflates OLS variance

This tradeoff is the fundamental reason regularization improves generalization in high-dimensional biomarker studies where p >> n.

When predictors are highly correlated, LASSO exhibits selection instability—it tends to arbitrarily select one feature from a correlated group and zero out the others. This behavior arises because the L1 penalty treats each coefficient independently and has no mechanism to enforce group inclusion. In biomarker discovery, this is problematic when:

• Multiple genes in a pathway are co-expressed and jointly predictive
• The selected feature may change across different data splits or bootstrap samples
• Important biological signals distributed across correlated markers are discarded

Elastic Net addresses this limitation by combining L1 and L2 penalties, encouraging grouped selection of correlated features while maintaining sparsity.

LASSO coefficients are typically computed using the LARS (Least Angle Regression) algorithm or coordinate descent. These methods efficiently generate the entire regularization path—the trajectory of all coefficient estimates as λ varies from infinity to zero—at the computational cost of a single OLS fit. Key properties of the solution path:

• Coefficients enter the model one at a time as λ decreases
• Once a coefficient becomes non-zero, it may later return to zero
• The path is piecewise linear, enabling exact computation of coefficient profiles

Modern implementations in glmnet (R) and scikit-learn (Python) use coordinate descent with warm starts and active set strategies for scalability to thousands of features.

In high-dimensional biomarker studies (e.g., genomics with p > 10,000 and n < 500), LASSO serves as a primary embedded feature selection method integrated directly into model training. Its application includes:

• Prognostic signature development: Selecting a sparse panel of genes predictive of survival outcomes using Cox-LASSO
• Diagnostic classifier construction: Building sparse logistic regression models that identify disease status with minimal biomarkers
• Multi-omics integration: Applying group LASSO variants to select relevant data modalities alongside individual features

A critical consideration: LASSO selects at most n features when p > n, which constrains biomarker panel size in small-sample studies. Post-selection inference methods like selective inference or stability selection are recommended to assess the reliability of selected biomarkers.