Lasso Regression

Lasso adds the absolute value of coefficient magnitudes as a penalty term to the ordinary least squares loss function. Unlike Ridge regression's L2 penalty, the L1 constraint creates a diamond-shaped constraint region that forces coefficients to hit the axes, producing exactly zero values for irrelevant features. The optimization objective is: minimize(Loss + λ * Σ|βᵢ|) where λ controls regularization strength.

The geometry of the L1 penalty causes continuous shrinkage and automatic variable selection simultaneously. As the regularization parameter λ increases, more coefficients are driven to exactly zero, effectively performing feature selection during model training. This eliminates the need for separate feature selection steps and produces a sparse model where only the most predictive features retain non-zero weights.

The sparsity property arises from the diamond-shaped constraint region of the L1 norm. At the optimal solution, the contours of the loss function touch the constraint region at corners, where some coefficients are exactly zero. This contrasts with Ridge regression's circular L2 constraint, which rarely produces zero coefficients. The result is a model with fewer non-zero parameters than observations.

The regularization parameter λ directly controls the bias-variance trade-off:

• Small λ: Low bias, high variance (approaches OLS, risk of overfitting)
• Large λ: High bias, low variance (simpler model, fewer features)
• Optimal λ: Typically selected via cross-validation to minimize prediction error

This makes Lasso particularly effective when the number of features exceeds the number of training samples.

In the LIME framework, Lasso serves as the sparse linear surrogate model trained on perturbed samples. Its ability to select only K most important features ensures the explanation remains human-interpretable. The non-zero coefficients directly indicate which features most influenced the black-box prediction in the local neighborhood, with the sign showing the direction of influence and magnitude indicating importance.

Key limitations include:

• Correlated features: Lasso arbitrarily selects one feature from a group of highly correlated predictors, potentially missing important relationships
• Non-convex extensions: Elastic Net combines L1 and L2 penalties to address the correlation limitation
• Feature scaling sensitivity: Input features must be standardized before applying Lasso, as the penalty is scale-dependent
• Selection consistency: Requires the irrepresentable condition for consistent feature selection in high dimensions

An interpretable surrogate model that uses Lasso regression or similar regularization to select only a small number of the most important features, ensuring the final explanation remains concise for human consumption. The L1 penalty forces coefficients of irrelevant features to exactly zero, creating a parsimonious model where only the most impactful variables survive. This sparsity is critical for instance-level explanations because humans can only process a handful of contributing factors at once. The resulting weight vector directly translates to a ranked list of feature contributions, with the sign of each coefficient indicating whether the feature pushed the prediction higher or lower.

A measure of how accurately an interpretable surrogate model approximates the behavior of the underlying black-box model in the immediate neighborhood of the instance being explained. Lasso regression balances fidelity against sparsity through the regularization strength hyperparameter (lambda):

• High lambda: Fewer features selected, lower fidelity, simpler explanation
• Low lambda: More features retained, higher fidelity, more complex explanation The goal is to find the sweet spot where the sparse linear model captures the local decision boundary without overfitting to the noise introduced by perturbation sampling.

The process of generating a synthetic neighborhood of data points by randomly altering or masking features of the original instance to create a training set for the local surrogate model. For tabular data, this involves drawing from feature distributions. For text, tokens are randomly removed. For images, superpixels are turned on or off. Each perturbed sample is then weighted by an exponential kernel based on its proximity to the original instance. The Lasso regression is trained on this weighted dataset, learning which features most strongly correlate with changes in the black-box model's predictions within the local region.

A class of explanation methods, including LIME and SHAP, that decompose a model's prediction into a sum of individual feature contributions relative to a baseline. Lasso regression naturally produces an additive explanation because the prediction of a linear model is simply the sum of each coefficient multiplied by its feature value. The final explanation takes the form: prediction = baseline + contribution_feature_1 + contribution_feature_2 + ... This additive property makes Lasso-based explanations directly comparable to Shapley values, though the underlying attribution philosophy differs—Lasso optimizes for local predictive accuracy while SHAP enforces game-theoretic fairness axioms.

The property that a local explanation should remain consistent across multiple runs with different random seeds, indicating that the identified important features are robust rather than artifacts of the sampling process. Lasso regression can produce unstable explanations when:

• The kernel width is too narrow, making the model sensitive to small sampling variations
• Features are highly correlated, causing the L1 penalty to arbitrarily select one from a group
• The number of perturbed samples is insufficient OptiLIME addresses this by automatically tuning the kernel width to maximize stability while maintaining acceptable fidelity, ensuring the sparse feature set is reproducible.

A human-understandable transformation of the original input features used as the input space for the surrogate model. For Lasso regression to produce meaningful explanations, raw features must first be converted into an interpretable domain:

• Tabular data: Continuous features are discretized into quantile bins
• Text: Raw tokens are converted to a bag-of-words presence vector
• Images: Pixels are grouped into superpixel segments that represent semantic regions The Lasso model is trained on these binary interpretable features, and the resulting non-zero coefficients directly map to human-understandable concepts like 'the word excellent was present' or 'the region around the eyes was visible.'