Student's T-Test

Unlike the Z-test, which assumes a normal distribution, the T-Test relies on the t-distribution. This distribution has heavier tails to account for the extra uncertainty introduced by estimating the population standard deviation from a small sample. As the degrees of freedom (sample size minus one) increase, the t-distribution converges to the standard normal distribution, making it robust for both small and large samples.

The T-Test has two primary variants critical for experimental design:

• Independent Samples T-Test: Compares the means of two distinct, unrelated groups (e.g., Control vs. Treatment users in a standard A/B test).
• Paired Samples T-Test: Compares means from the same group at two different times (e.g., pre-test vs. post-test scores). This variant controls for individual variability and is statistically more powerful for repeated measures.

The classic Student's T-Test assumes homoscedasticity (equal variances between the two groups). In real-world online controlled experiments, treatment effects often change the variance. Welch's T-Test is a safer, more robust alternative that does not assume equal variance. It adjusts the degrees of freedom using the Welch-Satterthwaite equation, preventing inflated Type I error rates when standard deviations differ between control and treatment.

To produce valid p-values, the T-Test requires:

• Continuous Data: The dependent variable must be interval or ratio scale (e.g., revenue per user, session duration).
• Independence: Observations must be independent of each other. Violations occur in social networks due to interference effects.
• Normality: The sampling distribution of the means must be approximately normal. Thanks to the Central Limit Theorem, this holds for large sample sizes even if the raw data is skewed.
• No Extreme Outliers: Extreme values can distort the mean and inflate the variance, masking true effects.

The T-Test can be configured directionally:

• Two-Tailed Test: Evaluates if the means are simply different. This is the standard conservative approach for A/B testing when a change could theoretically hurt or help the metric.
• One-Tailed Test: Evaluates if one mean is specifically greater or less than the other. This provides more statistical power but risks missing a significant effect in the opposite direction. It should only be used when a reverse effect is logically impossible or irrelevant.

The test statistic is a ratio of signal to noise: t = (Mean_Difference) / (Standard_Error_of_Difference)

• Numerator: The difference between the sample means (the 'signal').
• Denominator: The standard error of the difference (the 'noise'), calculated using the pooled or un-pooled variance. A larger absolute t-value indicates a greater difference relative to the variability in the data, leading to a smaller p-value and higher likelihood of rejecting the null hypothesis.

The probability of observing a test statistic at least as extreme as the one calculated, assuming the null hypothesis is true. It is the direct output of the T-Test calculation.

• Threshold: Commonly compared against an alpha of 0.05.
• Misinterpretation: It is not the probability that the null hypothesis is false.
• Calculation: Derived from the T-statistic and the degrees of freedom using the T-distribution's cumulative density function.

A parameter of the T-distribution that defines its exact shape, primarily determined by the sample size.

• Calculation: For an independent two-sample T-test, it is roughly n1 + n2 - 2.
• Impact: Lower degrees of freedom result in heavier tails, making it harder to reject the null hypothesis compared to a Z-test.
• Welch's Correction: The Welch's T-test uses a fractional, approximated degree of freedom to account for unequal variances.

The parametric assumption that the residuals or the sampling distribution of the mean is normally distributed.

• Robustness: The T-test is generally robust to violations of normality with large sample sizes (n > 30) due to the Central Limit Theorem.
• Diagnostics: Use Q-Q plots or the Shapiro-Wilk test to verify this assumption.
• Alternative: If the assumption is severely violated, use a non-parametric equivalent like the Mann-Whitney U Test.

An adaptation of the Student's T-Test that does not assume equal variances between the two groups.

• Default Choice: Generally preferred over the standard Student's T-Test in A/B testing because real-world user metric variances often differ between control and treatment.
• Mechanism: It adjusts the degrees of freedom downward to penalize the test when variances are unequal, reducing false positives.
• Comparison: The standard Student's T-Test uses a pooled variance estimate, which can be biased if group standard deviations differ significantly.

A standardized, scale-free measure of the magnitude of the difference between two group means.

• Calculation: The difference between the two means divided by the pooled standard deviation.
• Interpretation: 0.2 is small, 0.5 is medium, and 0.8 is large.
• Importance: A T-test can yield a statistically significant p-value with a trivially small effect size if the sample is huge. Effect size validates practical significance.

The error of rejecting the null hypothesis when it is actually true, controlled by the significance level (alpha).

• Context: Concluding a new personalization model increases conversion when it actually has no effect.
• Inflation: Running multiple T-tests simultaneously without correction (e.g., testing 20 metrics) drastically inflates the family-wise error rate.
• Mitigation: Use the Bonferroni Correction or control the False Discovery Rate (FDR) to adjust p-value thresholds.