Feature Importance

LIME decomposes a single prediction into a sum of individual feature contributions plus a base intercept. The feature importance score represents the additive effect of that feature on the prediction.

• Linear surrogate: For a linear model, the coefficient for feature x_i multiplied by its value is its contribution.
• Summation property: Prediction = Intercept + Σ(Feature_i_Contribution).
• Directionality: A positive score pushes the prediction toward a specific class; a negative score pushes it away.
• Magnitude: The absolute value indicates the strength of the influence, not just the presence of the feature.

Feature importance scores are strictly local—they explain only the specific instance being analyzed, not the model's global behavior. The same feature can have drastically different importance for different predictions.

• Instance-specific: An importance score is valid only in the immediate neighborhood of the explained data point.
• Decision boundary curvature: In highly non-linear regions, a feature might be critical for one prediction and irrelevant for a nearby one.
• Contrast with global methods: Unlike a single global feature importance ranking, LIME reveals that a model uses different reasoning paths for different inputs.
• Temporal stability: For the same instance, importance scores should remain stable across multiple runs with different random seeds.

To ensure human readability, LIME often employs Lasso (L1) regularization when training the surrogate linear model. This forces the coefficients of less important features to exactly zero.

• Concise explanations: Only the top K features with non-zero coefficients are presented, where K is a user-defined budget.
• Feature selection: L1 penalty performs automatic feature selection during surrogate training, eliminating noise.
• Trade-off: Higher regularization increases sparsity but may reduce local fidelity.
• Interpretable representation: The non-zero features are drawn from the interpretable representation (e.g., presence of a word, a superpixel), not raw pixels or token IDs.

The importance scores are derived from a locally weighted regression where perturbed samples closer to the original instance have higher influence on the surrogate model's fit.

• Exponential kernel: Sample weights decay exponentially with distance: weight = exp(-distance / kernel_width).
• Kernel width: This hyperparameter defines the effective neighborhood size. A small width enforces strict locality; a large width smooths the explanation.
• Distance metric: Varies by data type—cosine distance for text (TF-IDF vectors), Euclidean distance for tabular data, and superpixel overlap for images.
• Impact on importance: Features that are consistently predictive in the immediate vicinity receive high importance scores.

The feature importance scores reflect a fundamental compromise: a simple linear model may not perfectly capture the local decision boundary, leading to some loss of local fidelity.

• Fidelity measure: The R-squared score of the surrogate model on the perturbed samples indicates how well the importance scores represent the black-box behavior.
• Interpretability constraint: A linear model with sparse features is highly interpretable but may miss complex local interactions.
• Diagnostic tool: Low fidelity warns that the importance scores should be treated as an approximation, not a precise decomposition.
• OptiLIME: An automated method that selects the kernel width to balance fidelity and stability, ensuring the importance scores are both accurate and reproducible.

For image data, feature importance scores are visualized as saliency masks—heatmaps that highlight which superpixels contributed most to the prediction.

• Superpixel granularity: Importance is assigned to perceptually grouped pixel regions, not individual pixels, making the explanation semantically meaningful.
• Positive vs. negative evidence: Green regions indicate features supporting the predicted class; red regions indicate features contradicting it.
• Human validation: A domain expert can visually inspect the mask to verify if the model is focusing on clinically or operationally relevant regions.
• Explanation pipeline: The raw importance scores are mapped back to the original image coordinates to generate the final overlay.

An intrinsically interpretable model—typically a sparse linear model or shallow decision tree—trained to mimic the predictions of a complex black-box model within a tightly constrained local region. The surrogate's coefficients or splits directly encode feature importance for the specific instance being explained. The fidelity of this approximation is the primary constraint on the reliability of the resulting importance scores.

The process of generating a synthetic neighborhood of data points by randomly altering or masking features of the original instance. For tabular data, this involves drawing from feature distributions; for text, token masking; for images, superpixel occlusion. Each perturbed sample is weighted by its proximity to the original instance using an exponential kernel, creating the training set from which feature importance is derived.

A measure of how accurately the interpretable surrogate model approximates the black-box model's decision boundary in the immediate neighborhood of the instance. High local fidelity is essential for trustworthy feature importance scores. The metric is often quantified via R-squared for regression surrogates or weighted accuracy for classification, and is directly controlled by the kernel width hyperparameter.

A human-understandable transformation of raw input features that serves as the input space for the surrogate model. For images, this means superpixel segmentation—grouping pixels into perceptually meaningful atomic regions. For text, a bag-of-words or TF-IDF vector. The choice of representation fundamentally constrains which types of feature importance can be communicated to the end user.

An interpretable surrogate that uses Lasso regression (L1 regularization) to force the coefficients of less important features to exactly zero. This produces a concise explanation where only a small subset of features receive non-zero importance scores. The sparsity constraint directly addresses the fidelity-interpretability trade-off, ensuring the final explanation remains cognitively manageable for human review.

The property that feature importance scores should remain consistent across multiple runs with different random seeds. Unstable explanations—where important features vary dramatically between runs—indicate that the identified contributions are artifacts of the sampling process rather than genuine model behavior. OptiLIME and Bayesian LIME are frameworks specifically designed to optimize for this stability.