A/B Testing for ML

The fundamental structure of an A/B test. The control group (Group A) receives the current production model. The treatment group (Group B) receives the new candidate model. Users or requests are randomly and consistently assigned to one group to ensure a fair comparison. This random assignment is critical for isolating the effect of the model change from other variables.

The Overall Evaluation Criterion (OEC) is the single, pre-defined business or performance metric used to determine the winning variant. It must be:

• Aligned with business goals (e.g., click-through rate, conversion value, user retention).
• Statistically measurable from the experiment data.
• Pre-registered before the test begins to avoid bias.

Secondary metrics are also monitored to guard against regressions (e.g., latency, fairness).

Statistical power is the probability of correctly detecting a true effect (i.e., a real performance difference). It is determined before the test by calculating the required sample size. Key factors include:

• Minimum Detectable Effect (MDE): The smallest improvement you need to detect.
• Significance level (Alpha): The false positive rate (typically 5%).
• Baseline metric variance.

Insufficient sample size leads to underpowered tests that cannot reliably identify winners.

The system that assigns each incoming inference request to either the control or treatment model. It must guarantee:

• Consistent assignment: A user sees the same model version throughout a session to avoid confusing experiences.
• True randomness: Using a robust hashing function (e.g., on a user ID) to ensure groups are statistically comparable.
• Configurable split: Ability to adjust the percentage of traffic going to each variant (e.g., 50/50, 95/5 for a canary).

This is often implemented in the model serving layer or a feature flag service.

The mathematical process for determining if the observed difference in the OEC between groups is real or due to random chance. After the test runs, a hypothesis test (e.g., a t-test for means, chi-squared test for proportions) is performed.

• The result is a p-value. If p-value < alpha (e.g., 0.05), the result is statistically significant.
• Confidence intervals provide a range of plausible values for the true effect size.

Warning: Repeatedly "peeking" at results before the test concludes inflates the false positive rate.

A set of secondary metrics monitored to ensure the new model does not cause unintended harm. These act as circuit breakers. Common guardrails include:

• Performance Latency (p95, p99): The new model must not cause unacceptable slowdowns.
• Error Rates: Count of failed inferences or exceptions.
• Fairness Metrics: Performance across key user segments must not degrade.
• Business Metric Regressions: Other important metrics must not drop catastrophically.

A significant negative movement in a guardrail metric should trigger an automatic rollback or test halt.

A deployment strategy where a new model version is initially exposed to a small, controlled percentage of live traffic. This allows for performance monitoring and risk mitigation before a full rollout. It is a more gradual and controlled alternative to an immediate A/B test split.

• Key Mechanism: Traffic is routed using a weighted load balancer (e.g., 5% to new model, 95% to incumbent).
• Primary Goal: Detect critical failures or severe performance regressions with minimal user impact.
• Progression: A successful canary release often precedes a full A/B test to gather statistically significant performance data.

A zero-risk validation pattern where a new model processes live requests in parallel with the production model, but its predictions are logged and not returned to users. Also known as dark launching.

• Core Purpose: To gather performance metrics (latency, throughput) and prediction logs on real-world data without affecting the user experience.
• Validation Use Case: Ideal for detecting data drift or concept drift by comparing the new model's outputs against the production model's or ground truth (if available).
• Precursor to A/B Testing: Provides a safety check before exposing the model to any users in a canary or A/B test.

A serving architecture that enables a single, frozen base model (e.g., a large language model) to dynamically load different parameter-efficient fine-tuning (PEFT) modules (like LoRA adapters) per inference request.

• Efficiency for A/B Testing: Crucial for efficiently A/B testing multiple fine-tuned model variants. Instead of serving N full models, you serve 1 base model + N small adapter sets.
• Runtime Adapter Injection: The serving system injects the correct adapter weights into the model's computational graph at request time based on a routing key (e.g., experiment_group=B).
• Use Case: Allows cost-effective simultaneous testing of dozens of domain-specific model adaptations (e.g., finance vs. legal vs. support) using the same GPU resources.

The automated process of monitoring production model inputs and outputs to identify significant data drift (change in input feature distribution) or concept drift (change in the relationship between features and target).

• Relation to A/B Testing: Drift detection triggers the need for model retraining or adaptation. A/B testing is the methodology to validate that a newly trained model (adapted to the new data distribution) performs better than the old one.
• Statistical Methods: Employs tests like Population Stability Index (PSI), Kolmogorov-Smirnov, or model-based detectors to quantify drift.
• Continuous Monitoring: Runs alongside A/B tests to ensure the champion model's performance remains stable during the experiment.

A centralized repository for managing the lifecycle of machine learning models, including versioning, lineage, stage transitions (staging -> production), and metadata storage.

• Foundation for A/B Testing: The registry is the source of truth for the model versions (A and B) being tested. It stores:
  • Model binaries and PEFT adapter weights.
  • Training code, data snapshots, and hyperparameters for reproducibility.
  • Evaluation metrics from validation datasets.
• Orchestration Integration: The deployment/experimentation system (e.g., CI/CD pipeline) pulls specific model versions from the registry to deploy to the A and B inference endpoints.

Continuous Integration and Continuous Delivery/Deployment for Machine Learning extends software engineering practices to automate the testing, building, and deployment of ML models and data pipelines.

• Automates A/B Testing Rollouts: A robust CI/CD pipeline can automatically:
  1. Train a new candidate model (or adapter).
  2. Run validation tests.
  3. Deploy it as the 'B' variant to a staging endpoint.
  4. Initiate an A/B test with configured traffic splits.
• Gating Mechanism: Statistical significance from the A/B test can be a deployment gate in the CD process; only models that beat the baseline are promoted to full production.
• Tooling: Frameworks like Kubeflow Pipelines, MLflow, and TFX provide constructs for building these automated workflows.