A/B Testing Engine

The real-time computational core that selects the next best action, offer, or content variant for a specific user. It uses rules, predictive models, and contextual data to execute the logic defined by an A/B test.

• How it works: Ingests user context (e.g., session data, segment membership) and matches it against active experiment configurations.
• Key function: Determines bucket assignment (control vs. variant) and serves the correct experience without adding perceptible latency.
• Example: A decisioning engine evaluates a user's geolocation and referral source to decide whether to show the default hero image or a localized variant as part of an A/B test.

A software development mechanism that wraps a feature or content block in a conditional statement, allowing it to be toggled on or off remotely without deploying new code. It is the primary delivery vehicle for A/B test variants.

• A/B Testing Role: A feature flag gates access to an experimental code path. The flag's targeting rules define which user segments see the 'on' (variant) vs. 'off' (control) state.
• Benefits: Decouples deployment from release, enabling non-technical stakeholders to manage experiments and instantly roll back underperforming variants.
• Example: A flag called new-checkout-flow is set to serve the redesigned checkout to 10% of users in a canary release, which is a simple form of an A/B test.

The process of purging or updating outdated data from a cache when the source content changes. In dynamic assembly, it's critical to ensure users in an A/B test don't receive stale or incorrectly cached variants.

• The Challenge: A CDN might cache a page with Variant A. If a user assigned to Variant B requests the same URL, they must bypass the cache or trigger a purge to see the correct version.
• Solutions: Using Surrogate Keys to group and purge all cached fragments of a specific experiment, or leveraging Edge-Side Includes (ESI) to assemble uncached, personalized fragments at the edge.
• Example: When an A/B test concludes, a single API call purges all cached objects tagged with the experiment-123 surrogate key, instantly reverting all users to the winning variant.

A modular, self-contained piece of structured content—such as a headline, image, or call-to-action—designed for reuse and assembly. An A/B testing engine often swaps these fragments to create page variants.

• Atomic Testing: Instead of A/B testing entire pages, you test individual Content Fragments. This isolates the impact of a single element, like a headline, leading to clearer results.
• Structure: Each fragment includes its data (e.g., text) and metadata (e.g., author, schema type), making it semantically understandable for both the assembly engine and the testing logic.
• Example: A product detail page is assembled from fragments. An A/B test swaps the product-image fragment (a static photo) with a product-video fragment to measure the impact on conversion rate.

A mathematical measure confirming that the observed difference in performance between a control and a variant is likely not due to random chance. It is the definitive endpoint for an A/B test.

• The Metric: Typically expressed as a p-value. A p-value of less than 0.05 is a common threshold, indicating there is less than a 5% probability that the result is a false positive.
• Required Inputs: Calculation requires the conversion rate for each variant, the sample size (number of users), and the desired confidence level.
• Practical Use: An A/B testing engine continuously monitors these calculations and triggers an alert or automatically concludes the test when significance is reached, preventing premature decisions based on noisy data.

A reinforcement learning approach to A/B testing that dynamically shifts traffic away from underperforming variants toward better-performing ones in real-time, minimizing the opportunity cost of exploration.

• How it differs from A/B Testing: Traditional A/B tests split traffic evenly until a statistically significant result is found. A Multi-Armed Bandit algorithm continuously exploits the current best variant while still exploring others to a lesser degree.
• Key Benefit: Maximizes total reward (e.g., conversions) during the experiment itself, making it ideal for short-lived campaigns or high-traffic scenarios where the cost of showing a losing variant is high.
• Example: An e-commerce site uses a bandit algorithm to test four promotional banners. Within hours, the algorithm automatically directs 80% of traffic to the top-performing banner, while still sending 20% to the others to confirm the result.