A/B Testing

The foundation of any A/B test is the random division of traffic into two distinct groups. The control group receives the current version (A), while the treatment group receives the new variant (B). Random assignment is crucial to ensure groups are statistically equivalent, isolating the impact of the change being tested. For LLMs, groups might receive different prompts, different model versions (e.g., GPT-4 vs. Claude 3), or different post-processing logic.

Every valid test begins with a falsifiable hypothesis and a single, primary Key Performance Indicator (KPI). The hypothesis states the expected causal relationship (e.g., "Using a chain-of-thought prompt will increase answer accuracy by 5%"). The primary metric must be directly measurable, such as:

• Conversion Rate for a chatbot completing a task.
• Task Success Rate based on human evaluation.
• Average Session Duration or user engagement.
• Model Latency (P99) for performance tests.
• Hallucination Rate for accuracy tests.

Results must be statistically significant to be trusted. Statistical significance (typically p-value < 0.05) indicates that the observed difference between groups is unlikely due to random chance. Achieving this requires a sufficient sample size, calculated before the test begins based on:

• The minimum detectable effect (the smallest improvement you care to measure).
• The baseline conversion rate of the control.
• The chosen significance level (alpha) and statistical power (1-beta). Running a test without adequate sample size leads to inconclusive or false positive results.

This is the technical mechanism for routing users to the control or treatment variant. In modern LLM deployments, this is often managed by:

• Feature Flag Services (e.g., LaunchDarkly, Split) that expose runtime toggles.
• Service Meshes (e.g., Istio, Linkerd) capable of routing traffic based on headers.
• API Gateway configurations that direct requests to different backend model endpoints. Allocation is typically not 50/50; a common pattern is a 95/5 or 90/10 split initially, allowing for a canary analysis of the treatment's stability before expanding the test.

For LLM tests, maintaining session consistency is critical. A user must experience the same variant (A or B) throughout an entire conversational session to avoid a disjointed experience. This requires:

• Sticky Sessions: Using a user ID, session ID, or device ID to hash and consistently assign a user to the same group for the test duration.
• Context Propagation: Ensuring the assigned variant flag is passed through the entire request chain, from the API gateway through to the model inference service and any downstream agents or tools.

Post-test analysis involves more than just checking the primary KPI. Engineers must also monitor guardrail metrics to ensure the change hasn't introduced regressions. For LLM A/B tests, critical guardrails include:

• Inference Cost: Did the new prompt or model increase token usage?
• Error Rate: Are there more failed or timed-out requests?
• Safety/Compliance Violations: Did inappropriate output rates change?
• Secondary Engagement Metrics: Check for unintended impacts on other parts of the user journey. The decision to roll out the treatment is made only if the primary metric shows significant improvement and guardrail metrics remain stable.

A risk mitigation strategy where a new software version is initially released to a small, specific subset of users or servers. This allows teams to monitor performance metrics and error rates in a live environment with minimal impact before proceeding to a full rollout. It's a precursor to A/B testing, focused on stability rather than feature comparison.

• Key Mechanism: Traffic is routed based on user attributes, server instance, or a small percentage.
• Primary Goal: Validate stability and catch catastrophic failures early.
• Example: Releasing a new LLM inference engine to 5% of internal beta users first.

A runtime configuration mechanism that acts as a conditional toggle to enable or disable functionality without deploying new code. It decouples deployment from release, enabling:

• Trunk-based development: Engineers merge code to main behind flags.
• Instant rollback: Disabling a feature is immediate if issues arise.
• Granular targeting: Features can be enabled for specific user segments, which is foundational for A/B testing cohorts.

Flags are the enabling infrastructure that allows A/B tests to be conducted safely.

The routing layer that directs user requests to different backend service versions. It is the technical implementation behind A/B tests and canary deployments. Methods include:

• Percentage-based: 90% to Version A, 10% to Version B.
• Attribute-based: Route all users from a specific region to a new model variant.
• Consistent hashing: Ensures a user consistently sees the same version during a session.

This is managed by load balancers, API gateways, or service meshes like Istio or Linkerd.

An umbrella methodology for modern software release that combines techniques like canary deployments, feature flags, and A/B testing with continuous monitoring. The core principle is to gradually expose new features to users while measuring impact, allowing for rapid iteration or rollback.

• Contrast with A/B Testing: While A/B testing is a specific statistical comparison method, progressive delivery is the broader operational practice that often employs A/B testing as one of its tools.
• Feedback Loop: Relies heavily on Service Level Objectives (SLOs) and real-time observability to make go/no-go decisions.

A zero-risk validation technique where a new service version processes a copy of live production traffic in parallel, but its responses are discarded and not returned to users. This allows teams to:

• Compare performance: Measure latency and resource usage against the production version under identical load.
• Validate correctness: Check for differences in outputs (e.g., LLM responses) without user impact.
• Gather data: Pre-warm caches and collect logs before a real launch.

It's used before an A/B test to gain confidence in a new model's operational characteristics.

The mathematical cornerstone of valid A/B testing. It indicates that the observed difference in performance between variants (A and B) is unlikely due to random chance. Key concepts include:

• p-value: The probability of seeing the observed results if there was no real difference. A common threshold is p < 0.05.
• Sample Size: The number of observations needed per variant to detect a meaningful effect.
• Confidence Interval: A range of values that likely contains the true effect size.

Without achieving statistical significance, A/B test results are not reliable for decision-making.