Sample Ratio Mismatch

SRM detection relies on a Pearson's Chi-Squared test comparing observed user counts against expected counts. If you intend a 50/50 split but observe 50,100 in control and 49,900 in treatment, the test quantifies the probability this deviation is due to chance. A p-value below 0.001 typically triggers an automatic alarm, halting the experiment. This is not a metric evaluation; it is a pre-statistical infrastructure check that must pass before any analysis of lift or significance is valid.

SRM rarely indicates a problem with your model. It almost always signals a pipeline bug upstream of the analysis:

• Hashing Collisions: A non-uniform randomization seed causing deterministic bias in variant assignment.
• Residual Bot Traffic: Automated scrapers or click-farm traffic that bypasses your bot detection, artificially inflating one variant.
• Funnel Filtration: A downstream filter (e.g., 'active users only') that differentially removes users from one variant due to a performance bug.
• Telemetry Loss: A logging outage in one data center causing systematic data loss for a specific treatment group.

While related, these are distinct validation layers:

• SRM Check: A continuous, automated monitor running on every live experiment. It uses a simple count of randomized units to detect assignment failures.
• AA Test: A deliberate, time-bound experiment where both groups receive the identical control experience. It validates the entire metric computation pipeline, ensuring no false positives arise from variance in metric definitions. An AA test can pass while SRM fails if the assignment is broken but the metric logging is consistent.

Not all mismatches are fatal. The severity is a function of sample size and deviation magnitude:

• Tiny Sample: A 0.1% deviation on 1,000 users is likely noise.
• Massive Sample: A 0.1% deviation on 100 million users is almost certainly a systemic bug. The Chi-Squared statistic is sensitive to sample size, correctly flagging trivial percentage deviations as critical when the absolute user count is large enough to rule out random chance.

SRM can be an early symptom of a Stable Unit Treatment Value Assumption (SUTVA) violation. In social networks or ride-sharing marketplaces, if a treatment user invites a control user to the app, the control user's experience is contaminated. This 'interference' can manifest as a traffic imbalance because the treatment group is effectively 'generating' new users who are then incorrectly bucketed, skewing the observed ratio.

When the SRM alarm fires, the experiment is immediately quarantined. The standard triage workflow:

1. Freeze Traffic: Stop the experiment to prevent further data corruption.
2. Segment Diagnosis: Slice the mismatch by platform (iOS vs. Android), browser, or geography to isolate the faulty component.
3. Hash Audit: Verify the randomization seed and salt used by the experimentation library.
4. Bot Replay: Run the logged traffic through a secondary, stricter bot-detection model. Results from an SRM-flagged experiment are statistically unusable and must be discarded.

The primary statistical tool for detecting a Sample Ratio Mismatch. It compares the observed user counts in each variant against the expected counts based on the intended traffic allocation. A significant p-value (typically < 0.001) indicates a severe randomization failure. The test statistic is calculated as the sum of (Observed - Expected)² / Expected across all variants. A non-significant result does not guarantee the absence of bugs, but a significant result is a definitive stop signal.

An organizational metric monitored to ensure a new feature doesn't cause unintended harm. SRM is often the very first guardrail metric checked in an experiment. If the sample ratio is mismatched, all other metric calculations—such as conversion rates or revenue per user—are likely invalid due to systematic bias. Other guardrails include latency, crash rates, and gross merchandise volume.

The most common root cause of SRM. Bugs can occur at any stage of the data pipeline:

• Logging: Treatment events are dropped due to rate limiting or malformed payloads.
• ETL: A join key is missing, systematically filtering out users from one variant.
• Bot Filtering: Aggressive bot detection incorrectly classifies automated traffic from a specific treatment as invalid.
• Residual Data: Data from a previous experiment is not properly flushed, contaminating the new assignment.

A violation of the Stable Unit Treatment Value Assumption (SUTVA) where the treatment of one user influences the outcome of another. In two-sided marketplaces, a heavy discount treatment for riders might attract all available drivers, degrading the experience for the control group. This can manifest as an SRM if users in the control group churn or switch devices to gain access to the treatment, artificially inflating the treatment group's size.

A randomization technique that divides the population into homogeneous subgroups (strata) before assigning treatments. If stratification is implemented incorrectly—for example, using a faulty hash function for assignment—it can cause a Sample Ratio Mismatch within specific segments even if the global ratio appears correct. Properly implemented, it reduces variance and prevents covariate imbalance.

The statistical bias introduced when an experimenter checks interim results and stops the experiment early upon seeing a significant result. While SRM is a continuous monitoring check, it is not subject to the peeking problem in the traditional sense because it is a diagnostic for data integrity, not a primary metric. An SRM test should be run continuously from the first hour, and any significant result requires immediate investigation regardless of the experiment's planned duration.