Chi-Squared Test

The primary application in A/B testing is the Chi-Squared Test of Independence. It assesses whether two categorical variables—such as experiment variant (control vs. treatment) and conversion outcome (converted vs. not converted)—are statistically associated. The null hypothesis states that the variables are independent; rejecting it suggests the treatment had a significant effect on the categorical outcome.

The test statistic is calculated by comparing observed frequencies (actual user counts in each category) against expected frequencies (counts predicted if the null hypothesis were true). The formula is:

• χ² = Σ [ (O_i - E_i)² / E_i ] A large discrepancy between observed and expected values inflates the Chi-Squared statistic, indicating a lower probability that the difference is due to random chance.

Data for a Chi-Squared test is structured in a contingency table (or cross-tabulation). For a simple A/B test, this is typically a 2x2 matrix:

• Rows: Control Group, Treatment Group
• Columns: Success Event (e.g., Click), Failure Event (e.g., No Click) Each cell contains the frequency count of users falling into that specific combination of variant and outcome.

The degrees of freedom (df) define the shape of the Chi-Squared distribution used to calculate the p-value. For a test of independence, it is calculated as:

• df = (number of rows - 1) * (number of columns - 1) For a standard 2x2 A/B test, the degrees of freedom is 1. This parameter is critical for interpreting the test statistic against the correct theoretical distribution.

The validity of the Chi-Squared test relies on specific assumptions:

• Random Sampling: Data must be from a random sample.
• Categorical Data: Variables must be nominal or ordinal.
• Mutual Exclusivity: Each subject contributes to only one cell in the table.
• Expected Frequency Threshold: No more than 20% of expected cell counts should be below 5, and all individual expected counts should be at least 1. If this is violated, Fisher's Exact Test is a more appropriate alternative.

When applying the Chi-Squared test to a 2x2 contingency table, the discrete nature of the data can cause the continuous Chi-Squared distribution to overestimate significance. Yates' correction for continuity subtracts 0.5 from the absolute difference between observed and expected frequencies before squaring, providing a more conservative p-value and reducing the Type I error rate. Many modern experimentation platforms apply this correction by default.

The probability of observing a test statistic at least as extreme as the one calculated, assuming the null hypothesis is true. It is the primary threshold for determining statistical significance.

• A low p-value (typically < 0.05) suggests the observed data is inconsistent with the null hypothesis.
• It does not measure the probability that the null hypothesis is false or the magnitude of an effect.
• Misinterpretation often leads to the peeking problem and inflated false positives.

A reinforcement learning approach that dynamically allocates traffic to better-performing variants in real-time, minimizing opportunity cost.

• Unlike fixed-horizon tests, it balances exploration (testing new variants) with exploitation (sending traffic to the winner).
• Algorithms like Thompson Sampling or Upper Confidence Bound are used to make allocation decisions.
• Ideal for short-lifespan content like news headlines where learning must be rapid.

The probability that a test will correctly reject a false null hypothesis, representing the experiment's sensitivity to detect a true effect.

• Conventionally set at 80% (0.80), meaning a 20% chance of a Type II Error (false negative).
• Power is a function of sample size, effect size, and significance level.
• A power analysis is conducted before launch to determine the required traffic and duration.

A diagnostic check verifying if the observed traffic split matches the intended randomization ratio, serving as a primary infrastructure guardrail.

• A significant mismatch (e.g., a 50/50 split showing 60/40) indicates a critical bug in the randomization pipeline or data logging.
• Often caused by bot traffic, residual bias in hash functions, or faulty event triggering.
• This check should be automated and run continuously for every active experiment.

A time-series methodology that constructs a synthetic counterfactual baseline to estimate the causal effect of an intervention when a randomized control group is unavailable.

• Uses a Bayesian structural time-series model to predict what would have happened without the change.
• Essential for measuring the impact of marketing campaigns or model updates where holdouts are impossible.
• Relies on a set of control time series that were not affected by the intervention.

A long-term, stable subset of users permanently excluded from all experimental treatments to serve as a global baseline.

• Measures the aggregate, cumulative impact of all model changes over months or years.
• Prevents the "death by a thousand cuts" scenario where many small, positive experiments degrade long-term user experience.
• Essential for tracking the true North Star Metric without contamination from overlapping experiments.