Posterior Distribution

A conjugate prior ensures the posterior distribution belongs to the same parametric family as the prior, enabling closed-form updates without numerical integration.

• Beta-Binomial: A Beta prior updated with binomial data yields a Beta posterior
• Normal-Normal: A Normal prior for a Normal likelihood with known variance produces a Normal posterior
• Dirichlet-Multinomial: A Dirichlet prior updated with multinomial counts yields a Dirichlet posterior

Example: Starting with a Beta(2, 2) prior and observing 7 successes in 10 trials produces a Beta(9, 5) posterior. This analytical tractability made conjugate pairs essential before modern MCMC methods.

As sample size increases, the posterior distribution concentrates around the true parameter value, reflecting the diminishing influence of the prior.

• The Bernstein-von Mises theorem guarantees asymptotic normality of the posterior under regularity conditions
• The posterior variance scales as O(1/n), where n is the sample size
• The prior's influence decays proportionally to the ratio of prior effective sample size to actual data size

Example: With a weak prior equivalent to 2 pseudo-observations, after 1000 real observations the posterior is dominated by the likelihood, and the 95% credible interval shrinks by a factor of √500.

A credible interval directly quantifies posterior uncertainty by stating the probability that the parameter lies within a specified range, given the observed data.

• Equal-tailed interval: Excludes equal probability mass in both tails (e.g., 2.5% on each side for a 95% interval)
• Highest Posterior Density (HPD) interval: The narrowest interval containing the specified probability mass; every point inside has higher density than any point outside
• Interpretation: "There is a 95% probability the true parameter lies in [L, U]" — a direct probabilistic statement impossible in frequentist confidence intervals

Example: For a skewed posterior from an exponential likelihood with a Gamma prior, the HPD interval is narrower than the equal-tailed interval and better captures the region of highest plausibility.

The posterior predictive distribution averages the likelihood of future observations over the posterior distribution of parameters, propagating uncertainty into predictions.

• Formula: p(ỹ|y) = ∫ p(ỹ|θ) p(θ|y) dθ
• Accounts for both aleatoric uncertainty (noise in the likelihood) and epistemic uncertainty (uncertainty in θ)
• Overdispersion: Predictive intervals are wider than plug-in intervals that ignore parameter uncertainty

Example: In a Poisson count model, the plug-in prediction uses λ̂ = mean(y), but the posterior predictive incorporates uncertainty about λ, yielding a Negative-Binomial-like distribution with variance exceeding the mean.

Point estimates derived from the posterior minimize specific loss functions, linking Bayesian inference to decision-theoretic optimality.

• Posterior mean: Minimizes expected squared-error loss; E[θ|y]
• Posterior median: Minimizes expected absolute-error loss; robust to skew
• Posterior mode (MAP): Minimizes 0-1 loss for discrete parameters; penalized maximum likelihood interpretation
• Decision rule: δ(y) = arg min_a ∫ L(θ, a) p(θ|y) dθ

Example: For a highly skewed posterior from a variance parameter, the posterior median provides a more representative point estimate than the mean, which is pulled toward the heavy tail.

When conjugate priors are unavailable, the posterior is approximated using modern computational techniques that draw samples from the intractable distribution.

• Markov Chain Monte Carlo (MCMC): Constructs a Markov chain whose stationary distribution is the posterior; includes Metropolis-Hastings and Hamiltonian Monte Carlo (Stan, PyMC)
• Variational Inference (VI): Optimizes the parameters of a simpler distribution to minimize KL divergence from the true posterior; scales to massive datasets
• Laplace Approximation: Fits a Gaussian centered at the MAP estimate with covariance equal to the inverse Hessian of the log-posterior

Example: In a hierarchical model with thousands of random effects, VI provides a fast approximation while MCMC with the No-U-Turn Sampler delivers asymptotically exact posterior samples at higher computational cost.