Sparse Bayesian Learning

The foundational hierarchical prior that defines Sparse Bayesian Learning. ARD assigns an individual hyperparameter to govern the variance of each weight's prior distribution. During Bayesian inference, the data determines which hyperparameters tend toward infinity, forcing the corresponding weights to zero. This automatic pruning mechanism is what distinguishes SBL from standard L2 regularization, which only shrinks weights. The most common implementation uses a Gamma hyper-prior over the precision parameters, creating a Student's t-distribution marginal prior that is sharply peaked at zero with heavy tails—ideal for sparse solutions.

The canonical implementation of Sparse Bayesian Learning for regression and classification. The RVM is functionally identical to a Support Vector Machine (SVM) but operates within a fully probabilistic Bayesian framework. Key advantages over SVMs:

• Produces probabilistic predictions with full predictive distributions, not just point estimates
• Typically yields sparser solutions with fewer relevance vectors than support vectors
• Eliminates the need for cross-validation to set the C parameter or kernel hyperparameters
• Uses significantly fewer kernel functions at test time, reducing inference latency The RVM's likelihood function and ARD prior are combined via Type-II Maximum Likelihood to iteratively estimate hyperparameters.

The optimization framework used to estimate ARD hyperparameters without computationally expensive Markov Chain Monte Carlo (MCMC) sampling. Instead of integrating over all possible hyperparameter values, Type-II ML maximizes the marginal likelihood (model evidence) with respect to the hyperparameters. This analytically marginalizes over the weights, yielding a closed-form objective. The resulting optimization:

• Automatically balances data fit against model complexity
• Contains an implicit Occam's razor that penalizes unnecessary features
• Can be computed efficiently using the EM algorithm or direct gradient-based methods
• Provides a principled stopping criterion when hyperparameters converge

While both methods produce sparse solutions, their underlying mechanisms differ fundamentally:

• LASSO uses an L1 penalty as a convex relaxation of the L0 norm, solving a single optimization problem with a fixed regularization parameter λ
• SBL uses a hierarchical Bayesian model where sparsity emerges from the data through hyperparameter estimation
• LASSO's shrinkage is uniform across all coefficients; SBL adaptively shrinks each coefficient independently
• SBL provides full posterior distributions over weights, enabling uncertainty quantification that LASSO cannot offer
• In practice, SBL often produces sparser solutions than LASSO and is less prone to including spurious features when the true model is very sparse
• LASSO requires cross-validation to tune λ; SBL estimates all hyperparameters from the data directly

The sequential optimization algorithm introduced by Tipping and Faul (2003) that makes SBL computationally tractable for high-dimensional problems. Rather than optimizing all hyperparameters simultaneously, this method:

• Adds, removes, or updates a single basis function at each iteration
• Exploits matrix determinant and inverse lemmas to efficiently update the marginal likelihood
• Starts with an empty model and greedily adds relevant features
• Monitors the change in marginal likelihood to decide whether to add, re-estimate, or prune each basis function
• Achieves O(M³) complexity where M is the number of relevance vectors, not the total number of features This algorithm is what makes SBL practical for biomarker identification in wide genomic datasets.

The theoretical principle that prevents SBL from overfitting despite operating in high-dimensional feature spaces. The marginal likelihood (model evidence) automatically penalizes overly complex models because it integrates over all possible parameter values—a complex model spreads its probability mass thinly, reducing the evidence. This property:

• Embodies Bayesian Occam's razor: simpler explanations are preferred unless the data strongly supports complexity
• Provides a principled stopping criterion without needing a separate validation set
• Enables comparison of models with different numbers of features on equal footing
• Is computed analytically in SBL due to the conjugate Gaussian-Gamma prior structure
• Contrasts sharply with AIC or BIC, which are asymptotic approximations to the evidence