Differentially Private A/B Testing

Instead of computing exact means or conversion rates, the analyst injects calibrated noise drawn from a Laplace or Gaussian distribution into the aggregated metric. The noise scale is determined by the query's sensitivity—the maximum influence a single user can have on the statistic. This ensures the published lift or p-value reveals population-level trends without leaking individual treatment assignments or outcomes.

Traditional confidence intervals rely on exact sample variances, which can leak information. Private A/B testing frameworks construct empirical Bernstein confidence bounds that account for both sampling uncertainty and injected privacy noise. The resulting intervals are wider than non-private equivalents—a direct trade-off quantified by the privacy parameter ε—but provide statistically valid coverage guarantees without compromising individual data.

Continuous monitoring of experiments (peeking) compounds privacy loss. Private A/B testing employs composition theorems and privacy odometers to track cumulative ε expenditure across interim analyses. Analysts must pre-allocate a total privacy budget and decide on spending schedules—uniform, front-loaded, or threshold-based—before the experiment begins. Once the budget is exhausted, no further queries are permitted.

Standard differential privacy protects individual events or pageviews. In A/B testing, this is insufficient—a single user generates multiple events. User-level privacy bounds the contribution of an individual's entire session history. This is achieved by:

• Capping contributions: limiting each user to a maximum number of events or metric value
• Grouping by user ID before applying the differentially private mechanism This prevents attackers from inferring a specific user's assignment by observing multiple noisy metrics.

Before analyzing outcomes, experimenters verify that treatment and control groups are balanced on pre-experiment covariates. Private balance checks release differentially private histograms or private χ² test statistics for each dimension (country, device type, tenure). This prevents stratification attacks where an adversary infers individual group assignments by observing which covariates appear imbalanced in published summaries.

Analysts often segment results by subgroups (e.g., new vs. returning users) after seeing the data. This data dredging invalidates standard p-values and, under differential privacy, introduces additional leakage. Private post-selection inference frameworks apply noisy max mechanisms to select significant subgroups and adjust confidence intervals to account for both the selection process and the privacy noise, controlling the false discovery rate.

The privacy budget (ε) is the finite, quantifiable limit on total privacy loss allocated across all analyses in an A/B test. Composition theorems track cumulative loss when computing multiple metrics (e.g., mean, variance, p-value) on the same dataset.

• Sequential composition: Budgets sum linearly across queries
• Advanced composition: Tighter bounds using moments accountant
• Parallel composition: No budget cost for queries on disjoint user sets

A typical A/B test might allocate ε=1 for the primary metric and ε=0.5 for secondary metrics, with the total budget strictly enforced by the data access layer.

The Gaussian Mechanism is the primary noise injection method for differentially private A/B testing, adding calibrated noise drawn from a Gaussian distribution to continuous test statistics like the difference in means or t-statistics.

• Noise scale calibrated to L2 sensitivity of the statistic
• Achieves (ε, δ)-differential privacy with δ typically set to 1/n
• Enables confidence interval computation with adjusted variance

The mechanism ensures that the presence or absence of any single user's behavior in the treatment or control group cannot significantly shift the reported test outcome.

A/B tests must protect user-level privacy, meaning all events, clicks, or sessions belonging to a single user are treated as one privacy unit. This prevents attackers from inferring participation through repeated observations.

• Bounding contribution: Cap the number of events per user (e.g., max 100 clicks)
• User-level sensitivity: Query sensitivity defined by one user's maximum contribution
• Cross-event correlation: Protects against linkage across multiple metrics

Without user-level guarantees, an attacker could average multiple noisy observations of the same user to cancel out injected noise and reconstruct their behavior.

Traditional hypothesis tests (t-tests, chi-squared) must be adapted to account for injected noise. Differentially private hypothesis testing adjusts test statistics and critical values to maintain valid false positive rates.

• Noisy test statistic: T' = T + Noise, where Noise ~ N(0, Δ²/ε²)
• Adjusted null distribution: Variance inflated by noise variance
• Power analysis: Larger sample sizes required to overcome noise

A standard t-test with ε=1 on a metric with sensitivity Δ=0.1 requires approximately 10-20% more samples to achieve the same statistical power as its non-private counterpart.

When A/B test analysis is performed on a random subset of users rather than the full population, the privacy guarantee is amplified. The effective ε is reduced by a factor proportional to the sampling rate.

• Poisson sampling: Each user included independently with probability q
• Amplification bound: ε_effective ≈ q · ε_base for small q
• Trade-off: Reduced statistical power vs. stronger privacy

Sampling 10% of users for metric computation can amplify a base ε=1 guarantee to approximately ε=0.1, enabling more queries within the same total privacy budget.

Differentially private confidence intervals must account for both sampling variability and injected noise. Standard bootstrap methods leak privacy and must be replaced with privacy-preserving alternatives.

• Noise-aware intervals: Width = 1.96 · √(σ²_sampling + σ²_noise)
• Private bootstrap: Resampling with noise addition at each iteration
• Empirical Bernstein bounds: Concentration inequalities adapted for DP

A 95% confidence interval for a conversion rate difference with ε=0.5 might be ±2.3% compared to ±1.8% without privacy, reflecting the additional uncertainty from noise injection.