Prior Distribution

A prior is conjugate to a likelihood if the resulting posterior belongs to the same probability distribution family.

• Beta-Binomial: Beta prior + Binomial likelihood = Beta posterior
• Gamma-Poisson: Gamma prior + Poisson likelihood = Gamma posterior
• Gaussian-Gaussian: Gaussian prior + Gaussian likelihood = Gaussian posterior

This property enables closed-form updates without numerical integration, making inference computationally tractable.

Informative priors inject strong domain knowledge, concentrating probability mass on plausible parameter ranges. They accelerate convergence but risk bias if misspecified.

Non-informative priors (e.g., uniform, Jeffreys prior) aim to let the data dominate inference. Jeffreys prior is invariant under reparameterization, making it a principled default choice when prior knowledge is absent.

Many frequentist regularization techniques have exact Bayesian interpretations:

• L2 Regularization (Ridge): Equivalent to a Gaussian prior with zero mean on the weights
• L1 Regularization (Lasso): Equivalent to a Laplace prior, inducing sparsity

This duality reveals that regularization encodes an implicit prior belief that model parameters are small or sparse.

In hierarchical Bayesian models, the prior's own parameters (hyperparameters) are themselves assigned a prior distribution.

• Enables partial pooling across groups, sharing statistical strength
• Automatically learns the degree of shrinkage from the data
• Critical for multi-level modeling in domains like clinical trials and A/B testing

This structure prevents overfitting to individual groups while preserving group-level variation.

A validation workflow where you simulate data from the prior and likelihood before seeing real data.

• Reveals whether the prior encodes scientifically implausible outcomes
• Tests if the generative model produces reasonable effect sizes
• Essential for debugging complex probabilistic models

If simulated data violates domain constraints, the prior requires recalibration.

A pragmatic middle ground between fully informative and flat priors. These distributions are broad enough to let data dominate but bounded enough to regularize extreme values.

• Example: A half-Normal(0, 10) prior on a log-scale parameter
• Prevents the sampler from exploring unrealistic regions of parameter space
• Recommended default in modern probabilistic programming libraries like Stan

The workhorse of Bayesian modeling, placing highest probability density around a mean μ with decay controlled by variance σ².

• L2 Regularization equivalence: A zero-mean Gaussian prior on weights corresponds exactly to ridge regression
• Conjugate property: When combined with a Gaussian likelihood, the posterior is also Gaussian, enabling closed-form updates
• Use case: Default prior for regression coefficients, neural network weights in Bayesian neural networks, and latent factors
• Weakly informative variant: Use a wide variance (e.g., σ² = 100) to let the data dominate while maintaining proper integrability

A distribution with heavier tails than the Gaussian, placing more mass near zero and in the extremes simultaneously.

• L1 Regularization equivalence: A zero-mean Laplace prior on weights induces sparsity, corresponding to LASSO regression
• Sharper peak at zero: Encourages exact zero coefficients for irrelevant features, performing automatic feature selection
• Use case: High-dimensional regression where sparsity is expected, compressed sensing, and robust parameter estimation
• Scale parameter b: Controls the trade-off between sparsity and shrinkage strength

A flexible distribution defined on the interval [0, 1], parameterized by two shape parameters α and β.

• Conjugate to Bernoulli/Binomial likelihoods: Produces a Beta posterior, enabling straightforward sequential updating
• Shape control: α = β = 1 yields a uniform prior; α > 1, β > 1 concentrates mass around the mean; α < 1, β < 1 creates a U-shape favoring extremes
• Use case: Modeling conversion rates, click-through probabilities, and any bounded proportion parameter
• Natural interpretation: α - 1 can be viewed as pseudo-counts of prior successes, β - 1 as prior failures

The multivariate generalization of the Beta distribution, defined over the K-dimensional probability simplex.

• Conjugate to Multinomial/Categorical likelihoods: The posterior is also Dirichlet, with updated concentration parameters
• Concentration parameter α: A vector where each αᵢ controls the expected probability of category i; the sum α₀ = Σαᵢ controls overall dispersion
• Use case: Prior over categorical probabilities in topic models (Latent Dirichlet Allocation), mixture models, and evidential deep learning
• Symmetric Dirichlet: Setting all αᵢ equal to a value < 1 encourages sparse probability vectors where most mass concentrates in few categories

A distribution over positive real numbers commonly used as a semi-conjugate prior for variance parameters in normal models.

• Conjugate to Gaussian variance: When placed on σ² with a Gaussian likelihood, yields an Inverse Gamma posterior
• Shape α and scale β: Control the location and spread; small values (e.g., α = β = 0.001) create a vague prior approximating Jeffreys prior
• Use case: Hierarchical models specifying priors on variance components, Bayesian linear regression error terms, and random effect variances
• Caution: Can be inadvertently informative in the tails; the Half-Cauchy prior is often recommended as a more robust alternative for variance parameters

A global-local shrinkage prior designed for sparse signal recovery in high-dimensional regression.

• Mechanism: Each coefficient βᵢ gets a local shrinkage parameter λᵢ with a half-Cauchy prior, while a global shrinkage parameter τ controls overall sparsity
• Key property: Infinite spike at zero and heavy Cauchy tails allow strong shrinkage of noise while leaving large signals virtually unshrunk
• Use case: Genomic prediction with p >> n, image reconstruction, and any setting where most coefficients are near zero but a few are large
• Advantage over Laplace: Does not overshrink large coefficients, providing better separation between signal and noise

The updated belief about a parameter after observing data. It is computed by combining the prior distribution with the likelihood of the data via Bayes' theorem.

• Mechanism: Posterior ∝ Likelihood × Prior
• Role: Serves as the prior for future inference in sequential learning.
• Example: Starting with a weak prior about a drug's efficacy, the posterior tightens around the true effect after a clinical trial.

A prior distribution that, when combined with a specific likelihood function, yields a posterior distribution in the same parametric family.

• Purpose: Enables closed-form Bayesian updates without complex numerical integration.
• Classic Pair: A Beta prior with a Binomial likelihood produces a Beta posterior.
• Benefit: Drastically simplifies computation for real-time streaming analytics.

A neural network where weights are treated as probability distributions rather than fixed point estimates. The prior distribution over weights encodes initial assumptions about model complexity.

• Regularization: A Gaussian prior over weights corresponds to L2 regularization.
• Output: Produces a predictive distribution, enabling principled uncertainty quantification.
• Trade-off: Requires approximate inference (e.g., Variational Inference) due to intractability.

The objective function maximized in Variational Inference to approximate the true posterior. It balances data fit against the divergence from the prior distribution.

• Formula: ELBO = E[log Likelihood] - KL(Approx Posterior || Prior)
• Interpretation: The KL divergence term acts as a complexity penalty, keeping the posterior close to the prior.
• Usage: Essential for training Variational Autoencoders (VAEs).

Model uncertainty arising from a lack of knowledge about the true parameters. It is represented by the width of the prior distribution and is reducible with more data.

• Contrast: Distinct from Aleatoric Uncertainty (inherent noise).
• Behavior: High epistemic uncertainty far from training data; the posterior collapses to a point estimate with infinite data.
• Detection: Measured via Ensemble Disagreement or the variance of a BNN.

A non-informative prior distribution that is invariant under reparameterization. It is proportional to the square root of the determinant of the Fisher Information Matrix.

• Objective: Designed to have minimal influence on the posterior, letting the data speak.
• Property: Ensures that transforming the parameter space does not change the prior's meaning.
• Application: Used in objective Bayesian analysis when no subjective prior knowledge is available.