Sample Ratio Mismatches Invalidate A/B Test Results Through Uncorrectable Selection Bias

The foundation of online experimentation rests on a simple coin flip. When a technology company tests a new feature, it randomly assigns users to a control group or a treatment group, expecting the law of large numbers to make both populations identical. That initial randomization is the only mechanism that allows analysts to attribute any subsequent behavioral changes directly to the new feature.[5]

But the coin does not always land fairly. When the final tally of users in an A/B test diverges significantly from the intended allocation—such as a planned 50/50 split returning 50.2 percent in one arm and 49.8 percent in the other—the experiment suffers from a sample ratio mismatch.[1][2]

While a two-tenths of a percent deviation sounds trivial, at the scale of modern software, it represents a catastrophic failure. Across a sample of 1.64 million users, a 50.2 to 49.8 percent split is a statistical impossibility under fair randomization.[2]

"Sample ratio mismatch is evidence of selection bias," notes the experimentation platform Statsig in its 2023 technical documentation. If the assignment is not truly random, other hidden variables inevitably separate the two groups, destroying the causal link between the product change and the observed outcome.[5]

The phenomenon is remarkably common across the software industry. In a landmark 2019 paper presented at the Knowledge Discovery and Data Mining conference, researchers from Microsoft Corporation and Booking.com revealed that approximately 6 percent of all A/B tests conducted on their platforms exhibited a sample ratio mismatch.[3]

"I can recall hundreds of SRMs. We consider it one of the most severe data quality issues we can detect," one Microsoft analyst reported in the 2019 study. The researchers classified the mismatch not as a minor technical glitch, but as a fatal symptom of underlying data corruption that renders the resulting analysis completely untrustworthy.[3]

The Mechanism of Differential Attrition

To understand why a slight numerical imbalance destroys an experiment, analysts look to the mechanism that causes it. A sample ratio mismatch rarely occurs because the random number generator itself failed. Instead, it happens because users drop out of the experiment pipeline at different rates after the assignment occurs but before the telemetry logs their presence.[1][6]

Consider a software update that inadvertently adds 500 milliseconds of latency to a mobile application's loading screen. The randomization engine assigns users perfectly evenly at the server level, but users in the treatment group experience a noticeably slower application on their devices.[1]

Frustrated by the delay, a fraction of those treatment users close the application before the tracking pixel fires to record their participation. The control group, experiencing normal load times, logs its users reliably. The resulting data shows fewer users in the treatment arm, creating a classic sample ratio mismatch.[1][6]

This differential attrition means the users who vanished are not a random cross-section of the audience. They are specifically the users most sensitive to performance degradation. Because their data is missing entirely, the surviving treatment group looks artificially resilient, skewing the final conversion metrics upward.[6]

The Survivorship Bias Analogy

Microsoft researchers compare the phenomenon to the famous survivorship bias identified by statistician Abraham Wald during World War II. When the military asked Wald to optimize aircraft armor based on the bullet holes found on returning bombers, he famously recommended reinforcing the areas with no damage.[3]

Wald realized that the planes taking hits in critical areas simply never returned to be measured. In online experimentation, the users who encounter a broken feature, a redirect loop, or a bot-filtering error are the missing planes.[3]

"Just as Abraham Wald could not conduct a complete analysis of aircraft survival without considering those planes which did not return, so too do A/B testers need to be aware of missing users in their experiments," the Microsoft research team explained in their 2020 methodology review.[3]

When an experiment loses users disproportionately from one arm, the remaining population is fundamentally altered. Analyzing the survivors without accounting for the missing cohort guarantees a biased conclusion, often making a harmful product change appear highly successful.[3][7]

Why Reweighting Fails

When faced with a sample ratio mismatch, product teams often attempt to rescue the experiment by mathematically reweighting the data to force a 50/50 split. This approach assumes that the missing users would have behaved exactly like the surviving users, which is precisely the assumption that differential attrition violates.[2][4]

Data scientist Lukas Vermeer demonstrates the mathematical impossibility of correcting this bias through extreme-value bounding. If a control group logs 10,000 users while the treatment group logs only 9,000, the experiment is missing 1,000 counterfactual treatment users.[4]

To calculate the true bounds of the treatment effect, analysts must impute the best-case and worst-case scenarios for those 1,000 missing individuals. If the metric is a conversion rate, the worst-case scenario assumes zero conversions among the missing cohort, while the best-case scenario assumes a 100 percent conversion rate.[4][7]

Because the missing population is large enough to trigger the mismatch alert, imputing these extreme values invariably stretches the confidence interval across zero. The true treatment effect could be entirely negative or highly positive, proving mathematically that the point estimate is meaningless and cannot be salvaged.[4][7]

Detecting the Mismatch

Because eyeballing percentages is unreliable, experimentation platforms rely on strict statistical tests to flag imbalances. The industry standard is the Pearson's chi-square goodness-of-fit test, which calculates the probability that the observed distribution occurred by chance under a fair allocation.[1][5]

Platforms deliberately set the sensitivity threshold for this check much stricter than the standard 0.05 p-value used for reading experiment results. A 5 percent threshold would raise a false alarm on one in every twenty experiments, causing teams to ignore the warning entirely.[1]

Instead, systems like Microsoft's experimentation platform use a highly conservative threshold of p < 0.0005, while others standardize around p < 0.001. When an experiment trips an alarm at this level, the imbalance is almost certainly a genuine systemic fault rather than bad luck.[1][2]

The scale of the experiment heavily influences this calculation. In a small test with 1,000 users per arm, a 53 to 47 percent split might not trigger the chi-square threshold. However, at 1,000,000 users, a deviation of just 0.098 percent is statistically overwhelming and demands immediate investigation.[1][2]

Diagnosing the Root Cause

Once an experimentation platform flags a sample ratio mismatch, the analysis must halt. "Sample ratio mismatch is a stop sign for causal interpretation, not a cosmetic warning beside the result," experimentation consultant Atticus Li noted in a 2026 methodology brief.[2]

The diagnostic process requires tracing the user journey backward to find the exact point where the ratio breaks. Analysts must examine the assignment logs, the execution triggers, the telemetry pipelines, and the data filtering rules to isolate the leak.[2][6]

Common culprits include targeting rules that inadvertently exclude specific user segments from one variant, client-side execution failures caused by ad blockers, or bot-detection algorithms that aggressively filter out highly engaged users in a successful treatment arm.[3][6]

In one documented case at Microsoft, a treatment variant increased user engagement so dramatically that the platform's bot-detection system classified the genuine users as automated traffic and filtered them out. The resulting sample ratio mismatch initially made the highly successful feature look like a failure.[3]

The Financial Stakes

The financial stakes of ignoring these warnings are massive. In a case study known as the "$10 Million Mirage," a company ran an experiment that appeared to generate a 2 percent uplift in weekly revenue, projecting an annualized gain of $10 million.[8]

However, the experiment carried a subtle sample ratio mismatch, showing a 49.95 to 50.05 percent split. Upon investigation, the team discovered that internal employees had been inadvertently assigned exclusively to the treatment group.[8]

Because employees engaged with the product far more frequently than standard users, their presence artificially inflated the revenue metrics. Once the employee data was removed and the mismatch resolved, the actual incremental revenue dropped to zero, proving that a 1 percent imbalance can easily fabricate a 2 percent financial gain.[8]

Until the root cause is identified and the system is fixed, the data remains compromised. The only scientifically defensible response to an uncorrectable sample ratio mismatch is to discard the results, resolve the underlying instrumentation error, and launch a completely new experiment.[2][7]