Chi-Squared Test

The chi-squared test is fundamentally designed for categorical (nominal or ordinal) variables, not continuous data. It operates on frequency counts in a contingency table. Common applications include:

• Testing if gender is independent of product preference.
• Determining if the distribution of user sign-ups across regions matches expected marketing targets.
• Assessing if a model's error types are associated with specific input data categories.

The chi-squared goodness-of-fit test evaluates how well an observed frequency distribution matches an expected theoretical distribution.

Process:

1. Define a null hypothesis stating the observed data follows the expected distribution.
2. Calculate the test statistic (χ²) by summing the squared differences between observed and expected counts, divided by the expected counts.
3. Compare the statistic to a chi-squared distribution with the appropriate degrees of freedom.

Example: Testing if a six-sided die is fair by comparing the observed rolls of each number to the expected frequency of 1/6.

The chi-squared test of independence determines if there is a significant association between two categorical variables in a contingency table.

Key Mechanism:

• The null hypothesis states the variables are independent.
• Expected frequencies for each cell are calculated assuming independence: (row total * column total) / grand total.
• A large χ² statistic indicates the observed joint frequencies deviate substantially from what would be expected if the variables were unrelated, leading to rejection of the null hypothesis.

Use Case: In A/B testing, analyzing if the variant (A or B) is independent of a binary outcome like 'conversion' vs. 'no conversion'.

Chi-squared tests are large-sample approximations. They require sufficient expected frequencies in each contingency table cell to be valid. A common rule is that all expected counts should be at least 5. With sparse data (small samples or many categories), the test can be unreliable. In such cases, exact tests like Fisher's Exact Test are used instead. This requirement directly impacts experiment design, necessitating adequate traffic allocation in A/B tests involving categorical outcomes.

The shape of the reference chi-squared distribution is determined by degrees of freedom (df), which depend on the test type:

• Goodness-of-Fit: df = k - 1 - p, where k is the number of categories and p is the number of parameters estimated from the data.
• Test of Independence: For an r x c contingency table, df = (r - 1) * (c - 1).

The degrees of freedom are crucial for finding the critical value or p-value. A higher df results in a flatter, more spread-out chi-squared distribution, requiring a larger test statistic to achieve significance.

Interpretation: A significant result (p-value < alpha, e.g., 0.05) provides evidence against the null hypothesis but does not measure the strength or direction of the association. Effect size measures like Cramér's V or the phi coefficient must be calculated separately.

Key Limitations:

• Association, not Causation: The test identifies relationships but cannot prove cause-and-effect.
• No Directionality: It does not indicate which categories contribute most to the association (post-hoc analysis required).
• Sensitive to Sample Size: With very large samples, trivial associations may become statistically significant, necessitating business context for interpretation.