Deep Feature Selection

The defining architectural component is a sparse linear layer where each input feature is connected exclusively to its own weight node. This layer applies an elastic net regularization penalty (combining L1 and L2 norms) to these weights during training. Features whose corresponding weights are driven to exactly zero are effectively pruned from the model, achieving embedded feature selection without a separate preprocessing step.

Unlike filter or wrapper methods, DFS integrates selection directly into the learning process. The sparsity penalty is added to the standard loss function, and the entire network—including the selection layer—is optimized via stochastic gradient descent. This means feature selection and prediction are learned simultaneously, allowing the model to discover non-linear interactions among the selected features in subsequent hidden layers.

DFS employs a composite penalty on the input-to-first-hidden-layer weights:

• L1 penalty (LASSO): Encourages sparsity by shrinking individual weights to exactly zero.
• L2 penalty (Ridge): Stabilizes the solution and handles groups of correlated features. The balance between these penalties is controlled by a mixing hyperparameter, making DFS robust to multicollinearity while still producing a compact feature subset.

After the sparse selection layer, the network contains standard fully connected hidden layers with non-linear activation functions like ReLU or tanh. This architecture allows DFS to:

• Select features based on their contribution to complex, non-linear decision boundaries.
• Capture higher-order interactions between the surviving features in deeper layers.
• Outperform linear selection methods like LASSO when the underlying data relationships are inherently non-linear.

DFS occupies a unique position in the feature selection landscape:

• vs. LASSO: DFS can model non-linear relationships; LASSO is strictly linear.
• vs. Autoencoders: DFS produces an interpretable subset of original features; autoencoders create opaque latent representations.
• vs. Random Forest Importance: DFS learns selection and prediction jointly; tree-based importance is a post-hoc interpretation.
• vs. Recursive Feature Elimination: DFS is computationally more efficient for high-dimensional data, requiring only one training run.

DFS is particularly suited for biomarker identification in domains where the number of features (p) vastly exceeds the number of samples (n):

• Genomic selection: Identifying causal SNPs from genome-wide association data.
• Transcriptomics: Selecting gene expression signatures for cancer subtyping.
• Radiomics: Pruning thousands of imaging features to a clinically relevant subset. The built-in sparsity constraint naturally handles the curse of dimensionality common in biomedical datasets.

A deep learning architecture for unsupervised feature selection that uses a concrete relaxation of the discrete distribution in its bottleneck layer. Unlike Deep Feature Selection's one-to-one sparse linear layer, the Concrete Autoencoder learns a stochastic subset of the most informative input features by gradually annealing a temperature parameter during training.

• Key difference: Selects a discrete subset rather than applying continuous sparsity
• Mechanism: Uses the Gumbel-Softmax reparameterization trick
• Use case: When you need a hard, interpretable subset of original features

A linear regression method that performs both variable selection and regularization by adding a penalty equal to the absolute value of coefficient magnitudes. This shrinks some coefficients to exactly zero, effectively removing them from the model.

• Relationship to DFS: Deep Feature Selection generalizes this concept to neural networks by placing a sparse one-to-one layer at the input
• Limitation: LASSO is linear; DFS captures non-linear feature interactions
• Strength: Computationally efficient and highly interpretable

A robust method that combines subsampling with a high-dimensional selection algorithm like LASSO. Features are selected only if they are consistently chosen across many random data perturbations, providing strong control over false positives.

• Key concept: Selection probability threshold rather than coefficient magnitude
• Advantage over DFS: Provides formal false discovery rate control
• Integration: Can be applied on top of Deep Feature Selection outputs for added robustness

A model-agnostic technique that uses Shapley additive explanations to quantify each feature's contribution to predictions. Features with the highest global SHAP values are retained.

• Difference from DFS: SHAP is post-hoc and model-agnostic; DFS is embedded directly in training
• Advantage: Works with any black-box model
• Complementary use: Apply SHAP to interpret which features the Deep Feature Selection layer actually learned to use

A causal feature selection method that identifies the minimal set of variables making the target conditionally independent of all others. This set includes direct causes, effects, and confounders.

• Key distinction: DFS selects predictive features; Markov Blanket selects causally relevant features
• Output: Parents, children, and parents of children of the target node
• Use case: When you need to identify intervention targets, not just predictors

An extension of LASSO that performs variable selection on predefined groups of features, either selecting or discarding all members of a group together. This is particularly useful when features have a natural grouping structure.

• Contrast with DFS: Group LASSO requires pre-specified groups; DFS learns feature importance individually
• Mechanism: Uses an L2 norm within groups and L1 norm between groups
• Application: Gene pathway analysis where entire pathways should be selected together