Effect Size

The most widely used effect size for comparing the difference between two group means in standard deviation units.

• Interpretation: 0.2 = small, 0.5 = medium, 0.8 = large effect
• Formula: (Mean₁ − Mean₂) / Pooled Standard Deviation
• Use case: Comparing average order value between control and treatment groups in an A/B test
• Advantage: Unitless measure allows comparison across experiments with different outcome scales

A bias-corrected variant of Cohen's d that applies a correction factor for small sample sizes.

• Key distinction: Cohen's d slightly overestimates effect size when n < 20; Hedges' g corrects this
• Formula: Cohen's d × correction factor J
• When to use: Experiments with unequal or small group sizes where precision is critical
• Convergence: As sample size increases, Hedges' g and Cohen's d become nearly identical

An effect size measure that uses only the control group's standard deviation as the denominator, rather than the pooled standard deviation.

• Rationale: When treatment may affect variance, the control group SD provides a cleaner baseline
• Formula: (Mean₁ − Mean₂) / SD_control
• Primary use: Experimental designs where the treatment is expected to change variability, such as personalization algorithms that may polarize user behavior
• Conservative choice: Preferred when homogeneity of variance assumption is violated

Measures the strength and direction of a linear relationship between two continuous variables, ranging from -1 to +1.

• Interpretation: 0.1 = small, 0.3 = medium, 0.5 = large effect
• Squared value (r²): Represents the proportion of variance explained — a model with r = 0.5 explains 25% of outcome variance
• Application: Quantifying the relationship between a personalization score and conversion probability
• Limitation: Only captures linear relationships; use Spearman's ρ for monotonic non-linear associations

The proportion of total variance in the dependent variable attributable to a specific factor in ANOVA designs.

• Range: 0 to 1, interpreted as percentage of variance explained
• Benchmarks: 0.01 = small, 0.06 = medium, 0.14 = large effect
• Use case: Multi-variant A/B/n tests comparing more than two model variants simultaneously
• Partial η²: Used in factorial designs to isolate the effect of one factor while controlling for others

A business-facing effect size expressing the treatment's impact as a percentage improvement over the control baseline.

• Formula: ((Metric_treatment − Metric_control) / Metric_control) × 100%
• Example: Control conversion rate of 4.2% vs. treatment rate of 4.7% = 11.9% relative lift
• Critical caveat: Always report alongside absolute difference — a 50% lift on a 0.1% base rate is only 0.05 percentage points
• Best practice: Pair with Cohen's d for statistical rigor when communicating with technical stakeholders