Bayesian Inference

The prior quantifies existing knowledge or belief about a metric before observing new experimental data. In A/B testing for personalization, this could be a distribution based on historical conversion rates. A strong prior can act as a regularizer, preventing wild swings in estimates when sample sizes are small. Common priors include the Beta distribution for conversion rates and the Gamma distribution for count data. The choice of prior is a critical modeling decision that explicitly encodes assumptions into the analysis.

The likelihood represents the probability of observing the collected experimental data given a specific hypothesis about the model's performance. It acts as the objective function that scores how well different parameter values explain the results seen in the treatment and control groups. For binary outcomes like clicks, this is typically modeled using a Bernoulli distribution. The likelihood connects the abstract prior beliefs to the concrete evidence gathered during the experiment.

The posterior is the updated belief distribution calculated by combining the prior and the likelihood using Bayes' Theorem. Instead of a single point estimate like a p-value, the posterior provides a full probability distribution over the metric of interest, such as the lift in conversion rate. This allows direct probability statements like 'There is a 95% probability that Variant B is better than Variant A,' which are impossible in the frequentist framework. The posterior becomes the new prior for subsequent experiments.

A credible interval is the Bayesian counterpart to the frequentist confidence interval, but with a more direct interpretation. A 95% credible interval means there is a 95% probability that the true parameter value lies within that range, given the observed data and the prior. This is often what business stakeholders intuitively want from an A/B test result. The Highest Density Interval (HDI) is a specific type of credible interval that represents the narrowest band containing the specified probability mass.

A conjugate prior is a specific prior distribution that, when combined with a given likelihood, yields a posterior distribution of the same functional form. This mathematical property enables closed-form, computationally cheap updates without needing complex simulation methods like Markov Chain Monte Carlo (MCMC). Classic examples include:

• Beta-Binomial: For modeling conversion rates.
• Normal-Normal: For modeling average order value.
• Gamma-Poisson: For modeling purchase frequency. This efficiency is critical for real-time personalization engines that require instantaneous model updates.

Bayesian inference naturally accommodates sequential analysis, where data is evaluated as it arrives rather than waiting for a fixed horizon. The posterior from today's data becomes the prior for tomorrow's analysis, making it immune to the peeking problem that plagues frequentist tests. This allows product managers to continuously monitor experiments and stop them early if a clear winner emerges, without inflating false positive rates. This is foundational for dynamic traffic allocation in multi-armed bandit algorithms.

The classical statistical framework that derives conclusions from sample data by emphasizing the frequency or proportion of the data. Unlike Bayesian methods, frequentist inference relies strictly on p-values and confidence intervals without incorporating prior beliefs.

• Null Hypothesis Significance Testing (NHST) is the dominant paradigm
• Produces fixed point estimates rather than probability distributions
• Requires strict adherence to pre-planned sample sizes to avoid the peeking problem
• Commonly used in traditional A/B testing tools like Optimizely and Google Optimize

A reinforcement learning approach to experimentation that dynamically allocates more traffic to better-performing variants in real-time. This directly applies Bayesian principles by continuously updating posterior distributions over each arm's performance.

• Thompson Sampling: A Bayesian bandit algorithm that samples from the posterior distribution of each arm to decide traffic allocation
• Minimizes regret — the opportunity cost of showing inferior variants during testing
• Eliminates the need for fixed sample sizes and peeking corrections
• Ideal for high-velocity personalization where rapid iteration matters more than strict statistical rigor

A range of values likely to contain the true population parameter with a specified probability. In Bayesian A/B testing, this becomes a credible interval — a direct probabilistic statement about the parameter itself.

• Frequentist: "If we repeated this experiment 100 times, 95 of the intervals would contain the true value"
• Bayesian: "There is a 95% probability the true value lies within this range"
• Bayesian credible intervals are more intuitive for business stakeholders
• Can incorporate prior knowledge to narrow intervals when data is sparse

The probability of observing a test statistic at least as extreme as the one calculated, assuming the null hypothesis is true. Bayesian inference addresses the well-known misinterpretation of p-values by providing the posterior probability that a variant is better.

• P-value is NOT the probability that the null hypothesis is false
• Commonly misused threshold: p < 0.05 for "statistical significance"
• Bayesian alternative: "Probability variant B beats variant A is 87%"
• Eliminates the binary "significant/not significant" thinking that plagues frequentist testing

The statistical bias introduced when an experimenter repeatedly checks interim test results and stops early upon seeing a significant p-value. Bayesian inference naturally handles continuous monitoring because posterior probabilities are updated sequentially.

• Frequentist tests assume a fixed sample size determined in advance
• Each peek inflates the Type I error rate dramatically
• Bayesian methods allow "peeking" at any time without correction
• Enables always-valid inference — a key advantage for real-time personalization systems that need continuous decision-making

A quantitative measure of the magnitude of the difference between two groups, providing a standardized assessment of practical significance independent of sample size. Bayesian inference produces a full posterior distribution over the effect size rather than a single point estimate.

• Cohen's d: Standardized mean difference commonly used in A/B testing
• Bayesian approach shows the probability distribution of possible effect sizes
• Enables statements like "90% chance the lift is between 1.2% and 3.7%"
• Critical for assessing whether a statistically detectable difference is actually worth deploying