Intent-to-Treat Analysis

The cardinal principle of ITT analysis. It mandates analyzing all randomized participants in the groups to which they were originally assigned, regardless of protocol deviations, non-compliance, or crossover. This maintains the baseline comparability achieved by randomization, preventing selection bias that occurs if only compliant participants are analyzed. For example, in a drug trial, a participant assigned to the treatment group who never takes the medication is still analyzed as part of the treatment group.

ITT provides a pragmatic estimate of an intervention's effectiveness in real-world conditions, not its ideal efficacy under perfect adherence. It answers the question: "What is the effect of offering or assigning the treatment?" This includes the consequences of non-adherence and dropouts, which are realistic parts of any deployment. In contrast, a Per-Protocol analysis estimates efficacy under ideal conditions. For CTOs, ITT results are often more relevant for business decisions as they reflect expected real-world performance.

ITT analysis generally introduces a conservative bias toward the null hypothesis (i.e., no difference between groups). By including non-compliant participants in the treatment group, it dilutes the measured treatment effect. This is considered a strength in hypothesis testing, as it provides a more rigorous test. If a significant benefit is found under ITT, it is a robust finding. However, this bias is not guaranteed; in scenarios with non-compliance in the control group (e.g., control participants finding alternative treatment), the bias can be unpredictable.

A critical challenge for ITT is managing missing outcome data from participants who withdraw or are lost to follow-up. A true ITT analysis requires outcomes for all randomized subjects. Common strategies include:

• Last Observation Carried Forward: Using the last available measurement.
• Multiple Imputation: Statistically predicting missing values based on other data.
• Worst/Best-Case Scenario Analysis: Imputing extreme values to test result robustness. The chosen method must be pre-specified in the trial protocol to avoid bias. Simply excluding participants with missing data violates the ITT principle.

ITT is one of three primary analysis frameworks:

• Intent-to-Treat (ITT): Analyzes by original assignment. Preserves randomization, estimates effectiveness.
• Per-Protocol (PP): Analyzes only participants who perfectly followed the protocol. Estimates efficacy under ideal conditions but is prone to selection bias.
• As-Treated (AT): Analyzes participants by the treatment they actually received, regardless of original assignment. Severely breaks randomization and introduces major confounding. Modern trial reporting typically presents both ITT and PP analyses. A large discrepancy between them signals significant protocol non-adherence.

In technology A/B testing (e.g., comparing two AI models), the ITT principle is applied as analysis by assigned variant. A user who is assigned to see Model B but has a session error and sees nothing is still counted in Model B's results. This measures the true business impact of deploying Model B, accounting for real-world failures. Violating this—by analyzing only users who successfully received the treatment—can overstate benefits. ITT is crucial for accurate causal inference from online experiments, ensuring the estimated Average Treatment Effect is unbiased for the decision to launch.

Intent-to-treat analysis is a statistical principle that mandates analyzing all participants in a randomized controlled trial according to the group to which they were originally assigned, regardless of whether they actually received, adhered to, or completed the intended intervention. This preserves the randomization, which is the foundation for unbiased causal inference. In AI A/B testing, this means a user assigned to the treatment group (e.g., a new recommendation model) is analyzed as part of that group even if a system error prevented the model from loading for them, or if they dropped out of the session immediately.

• Purpose: To avoid selection bias that arises when analyzing only compliant participants.
• Contrast with Per-Protocol Analysis: ITT is conservative but unbiased; per-protocol analysis (analyzing only those who perfectly received the treatment) can introduce bias by comparing non-equivalent groups.

In online experiments comparing AI models, ITT analysis ensures the estimated treatment effect reflects the real-world impact of offering a new model, not just its ideal performance. Key scenarios include:

• Partial Feature Rollouts: If a feature flag fails for a subset of users assigned to the treatment, those users remain in the treatment group for analysis.
• Model Latency or Failures: Users who experience timeouts or errors from the new model are not moved to the control group; their outcomes (e.g., no purchase) are attributed to the treatment.
• User Attrition: Users who abandon an app during an experiment are included in the analysis, with their final observed state used (e.g., 'did not convert').

This approach measures the average treatment effect of deploying the model in a production environment with all its inherent imperfections, providing a realistic business impact estimate.

Understanding ITT requires contrasting it with alternative analytical approaches:

• As-Treated Analysis: Units are analyzed based on the treatment they actually received, not the one they were assigned. This breaks randomization and can severely bias results if the reasons for non-compliance are correlated with the outcome.
• Per-Protocol Analysis: A subset of as-treated analysis that only includes units that perfectly adhered to the experimental protocol. This maximizes internal validity for the 'perfect scenario' but sacrifices generalizability and introduces bias.

In AI, as-treated analysis might involve analyzing only the queries where the new model successfully returned a response. This could bias results if the model only succeeds on easier queries. ITT, by including all failures, gives a true picture of the system-wide effect.

Faithful ITT analysis depends on rigorous experiment logging. The system must persistently record two key facts for every user or request:

1. Assignment Variant: The group (Control A, Treatment B) determined by the initial deterministic hashing of a user ID during traffic splitting.
2. Final Outcome Metric: The business metric (click-through rate, conversion value, error rate) regardless of treatment delivery status.

This data must be immutable for analysis. Modern experiment platforms handle this by logging the assignment at the start of a user session and joining all subsequent events to this original assignment, ensuring the analysis aligns with the intent to treat.

ITT analysis is the gold standard for estimating the causal effect of a deployment decision. It answers the question: "What happens if we turn this feature on for everyone?"

• Causal Foundation: By preserving randomization, ITT provides an unbiased estimate of the Average Treatment Effect on the entire population.
• Informs SLOs/SLIs: The results, which include errors and latency, directly inform realistic Service Level Objectives for AI services. For example, if a new model increases 95th percentile latency by 200ms for 5% of users due to failures, ITT captures this degradation, whereas per-protocol analysis might miss it.
• Guardrail Metrics: ITT is essential for accurately monitoring guardrail metrics (e.g., system error rate) during a canary launch or A/B test.

Misapplying ITT can invalidate experiment results. Key pitfalls include:

• Analyzing Only Successful Exposures: A common error is filtering analytics to events where the 'treatment exposed' flag is true. This is an as-treated analysis, not ITT.
• The Peeking Problem: Repeatedly checking p-values before an experiment reaches its planned sample size inflates Type I error rates (false positives). While related to analysis timing, peeking is especially dangerous if combined with non-ITT filters, leading to premature and incorrect decisions.
• Ignoring Non-Compliance in Design: Not accounting for expected non-compliance (e.g., known integration failure rates) during statistical power and minimum detectable effect calculations can lead to underpowered experiments.

Proper ITT requires pre-registering the analysis plan and adhering to it, using methods like sequential testing with adjusted boundaries if early stopping is required.

Per-Protocol analysis is the methodological counterpoint to Intent-to-Treat. It analyzes only the subset of participants who fully adhered to the experimental protocol as assigned. This includes factors like:

• Receiving the full intended intervention
• Completing all required follow-up measurements
• Not crossing over to the other experimental group

While it estimates the efficacy of the treatment under ideal conditions, it is vulnerable to selection bias, as adherent participants often differ systematically from non-adherers.

The Average Treatment Effect is the primary causal quantity estimated in a randomized experiment. It is defined as the average difference in outcomes between the treatment and control groups across the entire population. In the context of ITT:

• The ITT analysis provides an estimate of the ATE.
• This estimate reflects the effectiveness of offering or assigning the treatment, not just its biological or algorithmic efficacy.
• It is often described as a policy-relevant effect, answering the question: "What happens if we deploy this model to everyone, knowing some will not use it correctly?"

Non-compliance refers to any departure from the assigned experimental protocol by participants. In AI A/B testing, this manifests as:

• Cross-over: Users assigned to Model A somehow receiving outputs from Model B.
• Non-adherence: Users ignoring or failing to trigger the new AI feature being tested.
• Missing Data: System failures causing loss of outcome measurements.

ITT analysis preserves the integrity of randomization by including these deviations in the analysis according to the original assignment, preventing bias from post-randomization confounding.

As-Treated analysis is a flawed but sometimes reported method where participants are analyzed based on the treatment they actually received, regardless of initial assignment. Key pitfalls include:

• It completely breaks the randomization, introducing severe confounding.
• Factors that influence whether a user actually receives the treatment (e.g., being more engaged) are often also related to the outcome.
• This analysis can produce misleading estimates of treatment effect. ITT is preferred for primary analysis to maintain internal validity.

A Randomized Controlled Trial is the gold-standard experimental design for establishing causality. Its core principles are:

• Random Assignment: Participants are allocated to treatment or control by chance.
• Control Group: Provides a counterfactual baseline.
• Blinding: Masking assignment from participants and/or evaluators to reduce bias.

Intent-to-Treat is the definitive analysis principle for RCTs. It ensures the estimate of the Average Treatment Effect is unbiased, as it compares groups as they were formed by randomization.

Selection bias occurs when the participants analyzed in a study are not representative of the population they are meant to reflect, due to systematic differences in selection. ITT directly combats a specific form of this:

• By analyzing all randomly assigned participants, ITT prevents bias from post-randomization selection (e.g., dropping non-compliant users).
• Alternative analyses like Per-Protocol are susceptible to this bias, as the "protocol-compliant" subset is a self-selected group that may be healthier, more tech-savvy, or otherwise atypical.