SHAP Feature Selection

SHAP values originate from cooperative game theory, where the Shapley value fairly distributes a payout among players based on their marginal contributions. In machine learning, features are the 'players' and the prediction is the 'payout.' The SHAP value for a feature is the weighted average of its marginal contribution across all possible feature subsets. This guarantees three desirable properties:

• Local Accuracy: The sum of SHAP values equals the model's prediction for an instance.
• Missingness: A feature with a missing value gets a SHAP value of zero.
• Consistency: If a model changes so a feature's contribution increases, its SHAP value does not decrease. This axiomatic foundation makes SHAP a uniquely principled method for feature attribution.

SHAP is a post-hoc, model-agnostic method, meaning it can explain any trained model, from gradient-boosted trees to deep neural networks. For feature selection, global importance is calculated by averaging the absolute SHAP values for each feature across all instances in a dataset. Features with the highest mean absolute SHAP values are retained. This approach captures complex, non-linear interactions that linear methods like LASSO miss. Common visualizations include:

• Summary Plot: Displays feature importance and the direction of each feature's effect.
• Dependence Plot: Shows how a feature's impact varies with its value and interacts with a second feature. This flexibility makes SHAP a universal tool for biomarker discovery across diverse model architectures.

Two primary implementations exist, optimized for different model types:

• KernelSHAP: A model-agnostic approximation that uses a weighted linear regression on a sample of feature coalitions. It is computationally expensive but works with any model. It requires a background dataset to represent missing features.
• TreeSHAP: An exact, polynomial-time algorithm specific to tree-based models (XGBoost, LightGBM, Random Forest). It calculates SHAP values directly from the tree structure without sampling, making it orders of magnitude faster than KernelSHAP. TreeSHAP also provides interaction values, decomposing a prediction into main effects and pairwise feature interactions. For high-dimensional genomic data, TreeSHAP on a gradient-boosted model is often the most computationally feasible approach.

A critical consideration in biomarker studies is that SHAP values, like permutation importance, can split importance among highly correlated features. If two genes are perfectly correlated, their SHAP values may be halved, potentially causing both to fall below a selection threshold. This is because SHAP evaluates marginal contributions, and a correlated feature adds little new information if its partner is already present. Mitigation strategies include:

• Hierarchical Clustering: Group correlated features first, then select a representative from each cluster.
• SHAP Interaction Values: Use TreeSHAP to identify and select the dominant feature in an interaction pair.
• Dimensionality Reduction: Apply PCA or autoencoders before SHAP analysis to decorrelate inputs. Understanding this behavior is essential to avoid discarding biologically valid but collinear biomarkers.

The computational cost of SHAP varies dramatically by implementation:

• KernelSHAP complexity is O(2^M * T * L), where M is the number of features, T is the number of samples, and L is the model inference time. For high-dimensional biomarker data with thousands of features, exact KernelSHAP is intractable. Sampling-based approximations are required, trading accuracy for speed.
• TreeSHAP complexity is O(T * L * D^2), where D is the maximum tree depth. This is linear with respect to the number of features, making it scalable to wide genomic datasets. For ultra-high-dimensional data (p > 10,000), a two-stage approach is common: use a fast filter method like Sure Independence Screening (SIS) to reduce dimensionality, then apply TreeSHAP for precise selection.

It is crucial to distinguish the dual role of SHAP in a biomarker pipeline:

• Feature Selection (Global): The goal is to reduce the feature space by selecting the top-k features with the highest mean absolute SHAP value across the entire training set. This produces a parsimonious, interpretable model for deployment.
• Model Explanation (Local): The goal is to explain a single prediction for a specific patient, showing which biomarkers drove that individual's risk score. This is critical for clinical trust and regulatory compliance. A common workflow is to first use SHAP for global selection to build a compact model, then use SHAP again on the final model to generate local explanations for clinicians. This dual application bridges the gap between model development and clinical translation.