Autoencoder Feature Extraction

The foundational architecture where the bottleneck layer has fewer neurons than the input layer, forcing the network to learn a compressed representation. The encoder maps input x to latent code h = f(x), while the decoder reconstructs x' = g(h). Training minimizes reconstruction loss L(x, x'), typically mean squared error or binary cross-entropy.

• Dimensionality constraint: Bottleneck size directly controls compression ratio
• Key insight: The network cannot learn the identity function, so it must capture salient structure
• Application: Initial dimensionality reduction for RNA-seq data with thousands of genes
• Limitation: No explicit regularization on the latent space structure

Imposes a sparsity constraint on the hidden layer activations, ensuring only a small fraction of neurons fire for any given input. This is achieved by adding a penalty term—either L1 regularization on activations or KL divergence between average activation ρ̂ⱼ and target sparsity ρ—to the loss function.

• Biological analogy: Mimics the sparse coding observed in the visual cortex
• Feature selectivity: Each hidden neuron learns to respond to a specific, interpretable pattern
• Hyperparameter: Target sparsity ρ (typically 0.05–0.1) controls activation frequency
• Use case: Extracting interpretable gene modules where each latent dimension represents a distinct biological pathway

Trains the network to reconstruct clean input from a corrupted version. During training, input x is stochastically corrupted to x̃ via noise (Gaussian, salt-and-pepper, or dropout), and the model learns to map x̃ → x. This forces the hidden layer to capture robust, high-level structures rather than memorizing pixel-level details.

• Corruption types: Additive Gaussian noise, masking noise (dropout), salt-and-pepper noise
• Key property: Learns the underlying data manifold, making representations invariant to small perturbations
• Stacked variant: Multiple DAEs can be stacked and fine-tuned for deep feature hierarchies
• Biomedical application: Robust feature extraction from noisy single-cell RNA-seq data with high dropout rates

Adds an analytic penalty on the Frobenius norm of the encoder's Jacobian matrix ||J_f(x)||²_F, which measures the sensitivity of the learned representation to small input perturbations. By minimizing this norm, the model contracts the input space locally, mapping nearby inputs to nearby latent representations.

• Mathematical formulation: Loss = L(x, g(f(x))) + λ ||J_f(x)||²_F
• Manifold learning: Explicitly encourages the representation to be invariant along the data manifold
• Key distinction: Unlike denoising autoencoders, the regularization is deterministic and analytic
• Advantage: Produces smoother, more stable feature representations than sparse or denoising variants

A generative probabilistic model that learns a smooth, continuous latent distribution rather than a deterministic encoding. The encoder outputs parameters of a distribution—typically mean μ and variance σ² of a Gaussian—and the latent vector z is sampled via the reparameterization trick: z = μ + σ ⊙ ε, where ε ~ N(0, I). The loss combines reconstruction error with KL divergence between the learned distribution and a prior (usually N(0, I)).

• Latent space structure: Enforces continuity and completeness, enabling meaningful interpolation
• β-VAE variant: Adds a β coefficient to the KL term to encourage disentangled latent factors
• Biomarker application: Generating synthetic patient latent profiles and identifying continuous disease trajectories
• Key advantage: The probabilistic framework provides uncertainty estimates for each latent dimension

An asymmetric architecture that masks a large fraction of input patches (e.g., 75%) and trains the decoder to reconstruct only the missing content from the visible patches and their latent representations. The encoder processes only the unmasked portions, dramatically reducing compute, while the lightweight decoder reconstructs the full input.

• Asymmetric design: Heavy encoder operates on ~25% of data; lightweight decoder reconstructs all patches
• Self-supervised paradigm: Learns rich representations without labels by solving the reconstruction task
• Scalability: Enables training on very large biomedical imaging datasets with reduced computational cost
• Application: Pre-training feature extractors on whole-slide pathology images before fine-tuning on diagnostic tasks

An unsupervised linear dimensionality reduction technique that transforms data into a new coordinate system of orthogonal principal components, ordered by the amount of variance they explain. Unlike autoencoders, PCA is limited to linear transformations and cannot capture complex non-linear manifolds. It is computationally efficient and deterministic, making it a common baseline before applying deep learning methods. The first few components often capture batch effects or technical noise in genomic data, requiring careful interpretation.

A generative variant of the autoencoder that learns a probabilistic latent space by encoding inputs as distributions (mean and variance) rather than fixed points. The KL divergence regularization term enforces a smooth, continuous latent structure ideal for generating synthetic patient data or interpolating between disease states. In biomarker discovery, VAEs provide a principled framework for handling the inherent stochasticity of biological measurements and enable uncertainty quantification in the learned representations.

An autoencoder variant that imposes a sparsity constraint on the hidden layer activations, forcing only a small fraction of neurons to fire for any given input. This mimics the sparse coding principles observed in biological neural systems and produces highly interpretable features. In high-dimensional biomarker data, sparse autoencoders can isolate distinct biological pathways into individual latent nodes, making them valuable for identifying specific disease mechanisms rather than entangled representations.

A robust autoencoder trained to reconstruct clean inputs from corrupted versions created by adding noise or applying dropout. This forces the network to learn stable, meaningful features rather than memorizing the input. For biomarker identification, denoising autoencoders are particularly effective with noisy biological assays like single-cell RNA sequencing, where technical dropout events are common. The learned representations are more resilient to measurement error and batch effects.

A deep learning architecture specifically designed for unsupervised feature selection rather than feature extraction. It uses a concrete relaxation of the discrete distribution in its bottleneck layer to learn a subset of the most informative original input features. Unlike standard autoencoders that create new latent variables, the concrete autoencoder directly identifies which original genes or biomarkers are most predictive, maintaining interpretability crucial for clinical validation and regulatory approval.

A group of algorithms that factorize a high-dimensional matrix into two lower-dimensional, non-negative matrices, yielding a parts-based, sparse representation. Unlike autoencoders, NMF produces purely additive combinations, which aligns naturally with biological processes where gene expression is inherently non-negative. This makes NMF particularly suited for extracting metagenes or molecular signatures that represent distinct cell types or pathway activities in tumor microenvironments.