Boruta

The core innovation of Boruta is the creation of shadow features—shuffled copies of the original features that serve as randomized negative controls. For each real feature, the algorithm creates a corresponding shadow feature by permuting its values across samples, destroying any relationship with the target while preserving the marginal distribution. The random forest is then trained on the combined set of real and shadow features, and the Z-score of each real feature's importance is compared to the maximum Z-score among all shadow features. A feature is deemed confirmed if its importance statistically outperforms the best randomized feature over multiple iterations.

Boruta operates through a multi-round elimination process using binomial statistical tests:

• In each iteration, features with importance significantly lower than the maximum shadow importance are marked as rejected and removed
• Features significantly higher are marked as confirmed
• Features in the grey zone remain tentative and continue to the next round
• The algorithm terminates when all features are either confirmed or rejected, or a specified iteration limit is reached
• The default significance level uses a two-sided binomial test with p < 0.01, adjusted for multiple comparisons across iterations

Unlike methods such as LASSO or Recursive Feature Elimination that seek a minimal-optimal set maximizing predictive accuracy, Boruta identifies the all-relevant feature set. This distinction is critical in biomarker discovery:

• A minimal-optimal set may discard features that are individually predictive but redundant with other selected features
• The all-relevant set retains every feature that carries information about the target, even if correlated with other predictors
• This property makes Boruta especially valuable for biological interpretation, where understanding all disease-associated variables matters more than building the most parsimonious model

Boruta leverages the random forest algorithm's native ability to capture non-linear relationships and feature interactions without parametric assumptions. The underlying importance metric is typically Mean Decrease in Accuracy (permutation importance) computed from out-of-bag samples. This foundation provides several advantages:

• Model-agnostic to data distributions—no assumptions of linearity or normality
• Automatically accounts for feature interactions through the ensemble of decision trees
• Handles mixed data types including continuous, categorical, and ordinal variables
• The randomness in bootstrap sampling and feature subset selection provides the variability needed for statistical comparison against shadow features

A distinctive aspect of Boruta is its handling of tentative features—variables that cannot be confidently classified as important or unimportant after the maximum iterations. The original algorithm offers two resolution strategies:

• TentativeRoughFix: Classifies all remaining tentative features as confirmed, erring on the side of inclusion for conservative biomarker discovery
• Tentative as rejected: A stricter approach that discards ambiguous features
• Modern implementations like BorutaShap replace the random forest importance with SHAP values, providing more consistent feature rankings and reducing the number of tentative features by using a more theoretically grounded importance metric

Boruta's exhaustive search approach carries computational costs that scale with dimensionality:

• Runtime complexity is approximately O(N * M * T * log(M)), where N is the number of iterations, M is the number of features, and T is the number of trees
• For datasets with thousands of features, runtime can extend to hours; the algorithm is best suited for datasets with hundreds to low thousands of features
• Parallelization across CPU cores can significantly reduce wall-clock time by distributing tree construction
• The BorutaPy Python implementation and Boruta R package both support multi-core execution and provide progress tracking for long-running selections

A foundational embedded method that performs simultaneous feature selection and regularization by penalizing the absolute size of coefficients. Unlike Boruta, LASSO selects a minimal predictive set rather than all relevant features.

• Shrinks irrelevant coefficients to exactly zero
• Produces a sparse, interpretable model
• Struggles with highly correlated feature groups
• Often used as a baseline before applying Boruta

A robust framework that applies a selection algorithm like LASSO across many random subsamples of the data. Features consistently chosen across perturbations are deemed reliable.

• Controls the expected number of false positives
• Complements Boruta's all-relevant philosophy
• Computationally intensive due to resampling
• Particularly valuable for small-n-large-p biomedical datasets

A statistical framework that creates synthetic 'knockoff' variables mimicking the correlation structure of original features. These serve as negative controls for rigorous false discovery rate estimation.

• Provides exact FDR control in finite samples
• Conceptually similar to Boruta's shadow features
• More formal statistical guarantees than Boruta
• Requires careful knockoff generation for complex data

A model-agnostic approach using Shapley additive explanations to quantify each feature's marginal contribution to predictions. Features with highest mean absolute SHAP values are retained.

• Based on cooperative game theory
• Provides both global and local interpretability
• Computationally expensive for large feature sets
• Often used post-Boruta for deeper importance ranking

A filter method that selects features by maximizing relevance to the target while minimizing mutual information redundancy among selected features. Unlike Boruta, it explicitly penalizes correlated selections.

• Uses information-theoretic criteria
• Fast and model-independent
• Produces a compact, non-redundant set
• Complements Boruta when interpretability demands minimal overlap

A model inspection technique that measures the decrease in model performance when a single feature's values are randomly shuffled. This breaks the feature-target relationship while preserving its distribution.

• Model-agnostic and intuitive
• Directly measures predictive impact
• Can be biased with correlated features
• Often compared against Boruta's shadow-based importance thresholds