T-Test

The Independent Samples T-Test (or two-sample t-test) compares the means of two separate, unrelated groups. It is the standard test for classic A/B testing scenarios where different users are randomly assigned to a control group (A) and a treatment group (B).

• Use Case: Determining if a new AI model version yields a different average click-through rate than the old version across two distinct user cohorts.
• Assumptions: Data in each group is approximately normally distributed, observations are independent, and variances between groups are equal (homoscedasticity). A Welch's t-test variant is used if variances are unequal.
• Example: Comparing the mean inference latency of Model A (n=100 users) versus Model B (n=100 different users).

The Paired Samples T-Test (or dependent t-test) compares the means of two related groups. It is used when measurements are taken from the same subjects under two different conditions, effectively analyzing the difference between paired observations.

• Use Case: Evaluating the performance of a single group of users before and after a model update, or comparing two models' outputs on the exact same set of input queries.
• Key Benefit: Controls for inter-subject variability by focusing on within-subject changes, often providing greater statistical power than an independent test.
• Example: Measuring the accuracy score of the same 50 diagnostic queries processed by a legacy model and a new model, then testing if the mean difference in scores is significantly different from zero.

The One-Sample T-Test determines whether the mean of a single sample group differs statistically from a known or hypothesized population mean.

• Use Case: Benchmarking a model's performance metric (e.g., a 95% accuracy score from a test set) against a predefined business target or industry standard (e.g., a 97% target).
• Mechanism: Tests the null hypothesis that the sample mean is equal to the specified value.
• Example: A company SLA requires model inference p99 latency to be ≤ 150ms. After 1000 inferences, the sample mean p99 is 155ms. A one-sample t-test evaluates if this observed 155ms is statistically greater than the 150ms target, accounting for sample variability.

Welch's T-Test is an adaptation of the independent samples t-test that does not assume equal variances between the two groups being compared. It is more robust when sample sizes or variances are unequal.

• Use Case: Comparing groups where variance homogeneity cannot be assumed, such as when a new AI feature is tested on a small, volatile user segment versus a large, stable one.
• Advantage: Provides a more reliable test when the equal variance assumption is violated, correcting the degrees of freedom to account for the disparity.
• Practical Note: Many modern statistical software packages default to or recommend Welch's test due to its robustness, making it a safer default for independent comparisons.

This distinction defines the directionality of the alternative hypothesis in any t-test.

• Two-Tailed Test: Used when you are testing for any difference between means, regardless of direction (e.g., Model A ≠ Model B). It splits the alpha significance level (e.g., 0.05) across both tails of the t-distribution. This is the standard, more conservative approach.
• One-Tailed Test: Used when you have a specific directional hypothesis (e.g., Model B > Model A). It places the entire alpha in one tail, making it easier to achieve statistical significance for that specific direction but offering no power to detect an effect in the opposite direction.
• Critical Choice: A one-tailed test should be justified by strong prior evidence or theory before seeing the data to avoid p-hacking.

Valid t-test inference relies on several statistical assumptions. Violations can lead to incorrect p-values.

Core assumptions include:

• Independence of Observations: Data points must not influence each other (violated by repeated measurements on same user without pairing).
• Approximate Normality: The sampling distribution of the mean should be normal. In practice, t-tests are robust to moderate deviations from normality, especially with larger sample sizes (n > 30 per group) due to the Central Limit Theorem.
• Homogeneity of Variance: For the standard independent t-test, groups should have similar variances. Use Levene's test to check or default to Welch's t-test.
• Scale of Measurement: Data should be continuous (or interval/ratio).

Remedy: For severe assumption violations, consider non-parametric alternatives like the Mann-Whitney U test (independent) or Wilcoxon signed-rank test (paired).

A t-test calculates a t-statistic by comparing the difference between two sample means to the variability within the samples. It assumes the data is approximately normally distributed and that the groups have similar variances (homoscedasticity). The resulting p-value indicates the probability of observing such a difference if the null hypothesis (no true difference) were true. Common variants include:

• Independent Samples t-test: Compares means from two separate, unrelated groups (e.g., Model A vs. Model B performance).
• Paired Samples t-test: Compares means from the same group at two different times or under two related conditions (e.g., model latency before and after an optimization).
• One-Sample t-test: Tests if a sample mean significantly differs from a known or hypothesized population mean.

The primary application in ML is to statistically validate that one model outperforms another. After training two models (e.g., a baseline and a new architecture), you evaluate them on a test set or via cross-validation, generating a set of performance scores (e.g., accuracy, F1-score) for each. An independent samples t-test can determine if the observed difference in average scores is statistically significant and not due to random chance in the test data split. This is more rigorous than simply comparing average metrics. For example, you might test: "Does our new BERT-based classifier have a higher mean accuracy than our old logistic regression model?"

T-tests are the workhorse for analyzing A/B tests (or split tests) in live AI systems. When comparing a new model variant (Treatment B) against the current production model (Control A), user interactions are randomly assigned. A key metric (e.g., click-through rate, conversion rate) is collected for each group. An independent samples t-test is then used to analyze the results, answering: "Is the difference in the mean metric between Group A and Group B statistically significant?" This provides a quantitative, probabilistic basis for launch decisions, moving beyond gut feeling. It directly ties to the null hypothesis that the two variants perform identically.

Beyond models, t-tests help validate the impact of individual features or data preprocessing steps. For instance, you can use a paired t-test to compare model performance scores before and after adding a new engineered feature, using the same cross-validation folds to control for variance. Similarly, you can test if the mean of a feature's values is significantly different between two classes (e.g., fraud vs. non-fraud transactions), which is a form of filter-based feature selection. This provides a statistical grounding for feature engineering decisions.

Misapplication of the t-test can lead to false conclusions. Key assumptions that must be checked include:

• Normality: The sampling distribution of the mean should be normal. With large sample sizes (>30 per group), the Central Limit Theorem often mitigates this.
• Homogeneity of Variance: The variances of the two groups should be equal. Welch's t-test is a robust variant that does not assume equal variances.
• Independence: Data points must be independent of each other (violated in time-series or clustered data).
• Random Sampling: Data should be a random sample from the population. A major pitfall is the multiple comparisons problem: running many t-tests without correction (e.g., Bonferroni) inflates the Type I error rate (false positives).

The t-test exists within a broader ecosystem of inferential statistics and experimental design:

• ANOVA: Extends the t-test to compare means across three or more groups.
• Confidence Intervals: Provide a range of plausible values for the true difference between means, offering more information than a binary significant/not-significant p-value.
• Statistical Power & Minimum Detectable Effect: Used in experiment planning to determine the required sample size for a t-test to reliably detect a meaningful difference.
• Non-Parametric Alternatives: The Mann-Whitney U test (for independent samples) or Wilcoxon signed-rank test (for paired samples) are used when t-test assumptions are severely violated.
• Bayesian Approaches: Methods like Bayesian estimation provide a posterior distribution for the difference, offering a more nuanced view than frequentist p-values.

Statistical significance is a determination that an observed difference between experimental groups is unlikely to be due to random chance. It is formally assessed by comparing a calculated p-value against a pre-defined significance level (alpha), commonly set at 0.05. A result is deemed statistically significant if the p-value is less than alpha, providing evidence to reject the null hypothesis of no effect. It is crucial to distinguish statistical significance from practical importance, as a tiny, trivial effect can be statistically significant with a very large sample size.

A p-value quantifies the probability of observing results at least as extreme as those in the experiment, assuming the null hypothesis (e.g., 'no difference between groups') is true. A low p-value (e.g., < 0.05) suggests the observed data is inconsistent with the null hypothesis.

• Not a Probability of Truth: It does not measure the probability that the null hypothesis is true or that the alternative hypothesis is false.
• Context-Dependent: Its interpretation depends entirely on the experimental design and sample size.
• Threshold Caution: Blind reliance on a 0.05 threshold without considering effect size and confidence intervals can be misleading.

A confidence interval provides a range of plausible values for an unknown population parameter (like the true difference between two means), estimated from sample data. A 95% confidence interval means that if the same experiment were repeated many times, 95% of the calculated intervals would contain the true parameter.

• More Informative than P-Value: It conveys both the estimated effect size and the precision of the estimate (a narrower interval indicates greater precision).
• Direct Interpretation: If a 95% CI for a difference in means is [1.5, 4.2], we are 95% confident the true difference lies between those values.
• Link to Significance: If a 95% CI for a difference excludes zero, it is equivalent to a statistically significant result at the alpha=0.05 level.

Statistical power is the probability that a test will correctly reject a false null hypothesis (i.e., detect a real effect). It is primarily influenced by sample size, effect size, and significance level.

The Minimum Detectable Effect is the smallest true effect size an experiment is powered to detect. Before running a t-test, practitioners must:

1. Define the desired power (typically 80% or 90%).
2. Set the significance level (alpha).
3. Estimate the expected variance in the data.
4. Calculate the required sample size to detect a practically meaningful MDE.

Underpowered experiments are a major pitfall, as they are likely to miss real effects, leading to false negatives.

A/B testing is a controlled online experiment methodology where two variants (A and B) of a system—such as different AI model versions, UI elements, or recommendation algorithms—are randomly assigned to users. The goal is to statistically compare their performance on a primary key performance indicator.

• Random Assignment: Uses deterministic hashing or random number generators for traffic splitting.
• Hypothesis-Driven: Starts with a clear null hypothesis (e.g., 'Model B's click-through rate <= Model A's').
• T-Test Application: An independent two-sample t-test is commonly used to analyze the results, comparing the mean metric value between the control (A) and treatment (B) groups.

A Multi-Armed Bandit is a sequential decision-making framework that dynamically allocates traffic between experimental variants. Unlike fixed-horizon A/B tests, it continuously balances exploration (gathering data on uncertain variants) with exploitation (favoring the currently best-performing variant).

• Adaptive Allocation: Traffic shifts toward better-performing options over time, minimizing opportunity cost during the experiment.
• Thompson Sampling: A popular Bayesian algorithm where an action is chosen by sampling from the posterior distribution of each variant's reward.
• Contrast with T-Test: While a standard t-test analyzes data after a fixed sample size, bandits make allocation decisions during the data collection process, optimizing for cumulative reward.