Bayesian Inference

The prior distribution represents pre-existing beliefs or knowledge about an unknown parameter before observing the current data. It is a foundational input to Bayes' theorem.

• Informative Priors: Encode specific, substantive knowledge (e.g., from historical data or domain expertise).
• Weakly Informative Priors: Regularize estimates without strongly influencing the result, helping with computational stability.
• Non-informative/Jeffreys Priors: Designed to have minimal influence, letting the data dominate the posterior.

In A/B testing, a prior could represent a belief about a new feature's baseline conversion rate based on historical performance of similar features.

The likelihood function quantifies the probability of observing the collected data given different possible values of the model's unknown parameters. It connects the parameters to the actual evidence.

• It is not a probability distribution over parameters but over data.
• The form of the likelihood is determined by the chosen data model (e.g., Bernoulli for clicks, Normal for continuous metrics).
• In an A/B test comparing two models, the likelihood for each variant models the observed success rates (e.g., clicks, conversions) given its true underlying performance parameter.

The posterior distribution is the central output of Bayesian inference. It represents the updated belief about the unknown parameters after combining the prior distribution with the observed data via the likelihood, according to Bayes' Theorem: Posterior ∝ Likelihood × Prior.

• It is a complete probability distribution, not a point estimate, encapsulating both the most probable value and the uncertainty around it.
• For an A/B test, the posterior for a treatment's conversion rate provides a full picture: the probable rate and the credible range of values.
• Decisions are made by analyzing this posterior (e.g., calculating the probability that Variant B is better than Variant A by at least 1%).

A credible interval is the Bayesian analogue to a frequentist confidence interval. It provides a range of values within which an unknown parameter lies with a specified posterior probability.

• Direct Probability Interpretation: A 95% credible interval means there is a 95% probability the true parameter value lies within that interval, given the data and prior. This is often the intuitive interpretation users mistakenly apply to confidence intervals.
• Highest Posterior Density (HPD) Interval: The most common type, representing the narrowest interval containing the specified probability mass.
• In reporting A/B test results, one might state: 'The posterior median lift is 2.4% with a 90% credible interval of [0.8%, 4.1%].'

Bayesian inference facilitates direct probability statements about hypotheses, enabling decision rules based on expected loss or probability thresholds.

• Probability of Superiority: The straightforward calculation from the joint posterior of P(Variant B > Variant A). If this probability exceeds a decision threshold (e.g., 95% or 99%), one may choose to deploy Variant B.
• Expected Loss: The expected detriment if a sub-optimal variant is chosen. A decision can be made to continue testing if the expected loss of deploying the currently best variant is still too high.
• This contrasts with frequentist null-hypothesis testing, which controls error rates in the long run but does not provide the probability that a specific hypothesis is true given the observed data.

A major operational advantage in live testing is that Bayesian methods allow for continuous monitoring and optional stopping without inflating false positive rates (the peeking problem).

• Because inference is based on the current posterior distribution, which incorporates all evidence up to that point, there is no statistical penalty for checking results early and often.
• This enables adaptive methods like Bayesian bandits, which can dynamically shift traffic toward better-performing variants while still learning about others.
• Teams can monitor a dashboard in real-time and make a launch decision as soon as the posterior probability of superiority crosses a predefined reliability threshold, optimizing both speed and confidence.

The mathematical engine of Bayesian inference is Bayes' Theorem: P(H|D) = [P(D|H) * P(H)] / P(D). This formula calculates the posterior probability P(H|D)—the updated belief in hypothesis H after observing data D. It combines the prior probability P(H) (initial belief) with the likelihood P(D|H) (probability of the data given the hypothesis). The denominator P(D) is the marginal likelihood, acting as a normalizing constant. This continuous update cycle is what enables adaptive learning from evidence.

Bayesian modeling explicitly defines three key distributions:

• Prior Distribution (P(H)): Encodes existing knowledge or assumptions about model parameters before seeing new data. For an A/B test click-through rate, a Beta distribution is a common conjugate prior.
• Likelihood Function (P(D|H)): Describes the probability of observing the collected data under a given hypothesis. For binary outcomes, this is often a Bernoulli or Binomial distribution.
• Posterior Distribution (P(H|D)): The result of Bayesian inference. It represents the complete updated belief about the parameters, combining the prior and the likelihood. We make probabilistic statements (e.g., 'There's a 95% probability variant B is better') directly from this distribution.

Bayesian and Frequentist (classical) inference offer fundamentally different interpretations of probability and experiment results.

Frequentist (Standard A/B Test):

• Probability = long-run frequency. Asks: 'If I ran this experiment infinitely, what would happen?'
• Outputs a p-value and confidence interval. Conclusion: 'We reject the null hypothesis that there is no difference.'
• Does not directly quantify the probability that one variant is better.

Bayesian:

• Probability = degree of belief. Asks: 'Given the data I observed, what is my updated belief?'
• Outputs a posterior distribution. Conclusion: 'There is a 92% probability that variant B has a higher conversion rate than A.'
• Allows for intuitive, direct probability statements about hypotheses.

In live experimentation, Bayesian methods provide a dynamic framework for decision-making.

• Real-time Updates: The posterior distribution updates continuously as new data arrives, allowing for optional stopping without inflating error rates (avoiding the peeking problem).
• Probabilistic Decisions: You can calculate the probability that B beats A directly from the posterior. A common decision rule is to declare a winner when P(B > A) > 95% or a Region of Practical Equivalence (ROPE) is defined.
• Incorporates Prior Knowledge: Historical data from past experiments can inform the prior, making new tests more efficient. For a completely new test, a weakly informative or uniform prior is used.
• Estimates Effect Size: The posterior provides a full distribution of the possible lift, not just a point estimate, enabling richer risk analysis.

Thompson Sampling is a quintessential Bayesian algorithm for the multi-armed bandit problem, which balances exploration and exploitation in adaptive experiments.

Mechanism:

1. For each variant (arm), maintain a posterior distribution for its reward rate (e.g., conversion).
2. On each new user visit, sample a single value from each variant's posterior distribution.
3. Serve the user the variant whose sampled value is highest.
4. Observe the outcome (click/no-click) and use it to update (Bayesian inference) that variant's posterior.

This naturally allocates more traffic to better-performing variants over time while still exploring uncertain ones. It is more efficient than fixed-percentage A/B tests for maximizing cumulative rewards during the experiment.

Calculating the posterior distribution can be analytically intractable for complex models. Modern Bayesian inference relies on computational techniques:

• Markov Chain Monte Carlo (MCMC): A class of algorithms (e.g., Gibbs sampling, Hamiltonian Monte Carlo) that draw sequential, correlated samples from the posterior distribution. Tools like Stan, PyMC, and TensorFlow Probability implement these methods.
• Variational Inference (VI): A faster, approximate method that frames inference as an optimization problem. It finds a simpler distribution (e.g., a Gaussian) that is closest to the true posterior. This is crucial for scaling to large datasets.
• Conjugate Priors: A special class where the prior and posterior are in the same probability family (e.g., Beta-Bernoulli, Gamma-Poisson). This allows for exact, closed-form posterior updates and is widely used in simple A/B testing models.

The mathematical formula that underpins all Bayesian inference. It describes how to update the probability of a hypothesis (H) given observed evidence (E): P(H|E) = [P(E|H) * P(H)] / P(E).

• P(H|E) is the posterior probability: our updated belief after seeing the evidence.
• P(H) is the prior probability: our initial belief before the evidence.
• P(E|H) is the likelihood: the probability of observing the evidence if the hypothesis is true.
• P(E) is the marginal likelihood or evidence, serving as a normalizing constant.

This theorem provides the precise mechanism for combining prior knowledge with new data.

A probability distribution that encodes existing beliefs or knowledge about an unknown parameter before observing the current data. Priors are a defining feature of the Bayesian framework.

• Informative Priors: Incorporate substantial domain knowledge (e.g., a normal distribution centered on a known average).
• Weakly Informative/Regularizing Priors: Gently constrain parameters to plausible ranges to stabilize estimation (e.g., Normal(0,10) on a regression coefficient).
• Non-informative Priors: Designed to have minimal influence, letting the data dominate (e.g., a uniform distribution). Improper priors, like a uniform distribution over an infinite range, are sometimes used but require care.

The choice of prior is explicit, making modeling assumptions transparent.

A function that measures the plausibility of the observed data given different possible values of the model's parameters. It is central to both Bayesian and frequentist statistics.

• For data D and parameters θ, the likelihood is L(θ | D) = P(D | θ).
• It is not a probability distribution over θ but over the data. In Bayesian inference, it is used to re-weight the prior distribution.
• The principle of maximum likelihood (frequentist) estimates parameters by finding the values that maximize this function.
• Common forms include Gaussian (for continuous data), Bernoulli (for binary outcomes), and Poisson (for count data).

The end result of Bayesian inference: a probability distribution that represents updated beliefs about the unknown parameters after combining the prior distribution with the observed data via the likelihood.

• It is the solution: Posterior ∝ Likelihood × Prior.
• Contains all probabilistic information about the parameters given the data.
• Summaries like the posterior mean, median, or mode provide point estimates.
• Credible Intervals (e.g., 95%) can be derived directly from the posterior, allowing statements like "There is a 95% probability the parameter lies in this interval."
• Computing the posterior often requires techniques like Markov Chain Monte Carlo for complex models.

A class of computational algorithms for sampling from a probability distribution, most famously used to approximate posterior distributions in Bayesian inference when they cannot be calculated analytically.

• MCMC constructs a Markov chain that has the desired posterior distribution as its equilibrium distribution.
• After a burn-in period, samples from the chain are used as approximate samples from the posterior.
• Key algorithms include:
  • Metropolis-Hastings: A general-purpose algorithm that proposes new parameter values and accepts/rejects them based on a probability ratio.
  • Gibbs Sampling: Iteratively samples each parameter conditional on the current values of all others, efficient when conditional distributions are known.
• Tools like Stan, PyMC, and TensorFlow Probability implement advanced MCMC variants.

A prior distribution chosen such that the posterior distribution belongs to the same probability family as the prior. This simplifies computation dramatically, as the posterior can be derived analytically.

• Conjugacy provides a closed-form solution, bypassing the need for MCMC in simple models.
• Classic examples:
  • Beta prior + Binomial likelihood → Beta posterior (for proportion data).
  • Gamma prior + Poisson likelihood → Gamma posterior (for rate data).
  • Normal prior + Normal likelihood (with known variance) → Normal posterior (for mean estimation).
• While computationally convenient, conjugate priors are sometimes chosen for mathematical tractability rather than accurately representing prior knowledge.