Recursive Feature Elimination (RFE)

RFE operates through a recursive pruning process that systematically removes the least informative features. The algorithm:

• Trains the chosen estimator on the current feature set
• Ranks features by importance via coefficient weights or feature importance scores
• Eliminates the lowest-ranked feature or a specified step size of features
• Repeats on the reduced set until the desired number of features remains This wrapper approach evaluates feature subsets in the context of the specific model, capturing feature interactions that filter methods miss.

The elimination order depends entirely on the base estimator's intrinsic importance metric:

• Linear models: Absolute coefficient magnitudes (|w_i|) from coef_
• Tree-based models: Gini importance or mean decrease in impurity from feature_importances_
• SVM with linear kernel: Squared coefficient weights from the separating hyperplane
• Any model with a coef_ or feature_importances_ attribute can serve as the RFE estimator This tight coupling means RFE inherits the biases of the underlying model—a linear estimator will miss non-linear relationships.

RFECV extends standard RFE by automatically determining the optimal number of features through cross-validation:

• Performs RFE to generate ranked feature subsets of all possible sizes
• Evaluates each subset size using k-fold cross-validation on the training data
• Selects the smallest feature set within one standard error of the peak performance score
• Eliminates the need to manually specify n_features_to_select This variant guards against overfitting by balancing predictive performance with model parsimony, making it essential for high-dimensional biomarker discovery.

The step parameter controls how many features are removed per iteration, creating a speed-accuracy trade-off:

• step=1: Exhaustive search removing one feature at a time—most precise but computationally expensive for high-dimensional data (O(n_features²) model fits)
• step > 1: Removes multiple features per iteration—faster but risks eliminating correlated features that are individually weak but jointly informative
• step as float (0.0–1.0): Removes a percentage of remaining features per iteration, accelerating convergence on wide datasets For genomic datasets with 10,000+ features, a larger step size or a pre-filtering stage is often necessary.

RFE exhibits selection instability when features are highly correlated, a common scenario in biomarker identification:

• Two perfectly correlated predictors will have arbitrary ranking order, as removing one leaves the other to capture the full effect
• Small data perturbations can cause dramatic reordering of correlated feature groups
• The final subset may exclude biologically relevant markers simply because a correlated proxy was retained Mitigation strategies include using stability selection wrappers, applying RFECV with multiple random seeds, or pre-clustering correlated features before elimination.

The sklearn.feature_selection module provides two primary classes:

• RFE(estimator, n_features_to_select): Basic recursive elimination with manual specification of the target feature count
• RFECV(estimator, cv, scoring): Cross-validated variant that automatically selects the optimal subset size Key attributes after fitting include:
• ranking_: Integer rank for each feature (1 = selected, higher = eliminated earlier)
• support_: Boolean mask of selected features
• grid_scores_ (RFECV only): Cross-validation scores for each subset size For biomarker discovery workflows, RFECV with a linear SVM or random forest is a common starting point.

An embedded method that performs feature selection and regularization simultaneously by adding a penalty equal to the absolute value of coefficient magnitudes. Unlike RFE, LASSO shrinks irrelevant coefficients to exactly zero during model fitting rather than iteratively removing them.

• Produces a sparse model in a single training run
• Handles the p > n problem common in genomics
• Selection is tied to the linear model's objective function

An all-relevant wrapper method that extends random forest importance scores by comparing real features against randomly shuffled shadow features. Unlike RFE's greedy elimination, Boruta identifies all features with any meaningful relationship to the target.

• Creates shadow attributes as null distribution baselines
• Iteratively removes features confirmed irrelevant
• Captures non-linear interactions that linear RFE may miss

A robust framework that applies a selection algorithm like LASSO to many random subsamples of the data and retains features consistently chosen across perturbations. Addresses RFE's sensitivity to small data fluctuations.

• Controls the expected number of false positives
• Provides a selection probability per feature rather than a binary decision
• Particularly valuable for biomarker discovery where reproducibility is critical

A model-agnostic approach that uses Shapley additive explanations to quantify each feature's marginal contribution to predictions. Unlike RFE's reliance on internal model weights, SHAP values provide consistent importance measures across any model type.

• Based on cooperative game theory principles
• Handles feature interactions transparently
• Works with tree ensembles, neural networks, and kernel methods

A causal feature selection method that identifies the minimal set of variables making the target conditionally independent of all others. This set includes direct causes, effects, and confounders—providing stronger guarantees than RFE's correlational approach.

• Discovers the true causal neighborhood of the target
• Eliminates redundant but non-causal predictors
• Uses conditional independence tests rather than model importance scores

A statistical framework for controlled variable selection that creates synthetic 'knockoff' variables mimicking the original features' correlation structure. These serve as negative controls, enabling rigorous false discovery rate estimation that RFE cannot provide.

• Guarantees exact FDR control under minimal assumptions
• Constructs paired null variables for each real feature
• Widely adopted in genome-wide association studies