How the Minimum Detectable Effect, Statistical Power, and Alpha Determine the Required Sample Size for an A/B Test

Product managers and marketers frequently claim that running an A/B test simply requires waiting until a dashboard shows a statistically significant result, treating sample size as a flexible afterthought. However, the mathematical reality of experimental design contradicts this approach. The required sample size is a rigid mathematical output determined before the test even begins, dictated by three interlocking constraints: the Minimum Detectable Effect (MDE), statistical power, and the significance level (alpha).

If an experimenter launches a test without fixing these three dials in advance, they risk running an underpowered test that produces false negatives or illusory winners that vanish upon full deployment. As Wikipedia's statistical documentation notes, power is a function of the specific test used, the sample size, and the effect size [5].[5]

The first and most sensitive dial is the Minimum Detectable Effect. MDE represents the smallest true effect size that a study can detect with a specified probability given the available sample size [1]. It acts as a business decision translated into mathematics: what is the smallest conversion rate change that would justify the engineering cost of implementing the winning variant?[1]

Many teams overestimate the effect size they expect to see, which is a primary cause of underpowered tests. Because MDE scales with the inverse square root of the sample size, the penalty for seeking smaller effects is severe [1]. According to MetricGate's testing documentation, halving the MDE requires four times the sample size [1].[1]

The second dial is statistical power, which acts as the experiment's defense against false negatives, or Type II errors. Power is the probability of finding a difference between the performance of the control and variant, assuming a true difference actually exists [3].[3]

SplitMetrics notes in its 2023 testing manual that a power level of 0.80 is considered standard best practice across the industry [3]. This means there is an 80% chance of the test identifying a significant difference if one exists, leaving a 20% risk that a genuine improvement will go undetected.[3]

Some data science teams push for 90% power when missing a true effect would be financially costly, though this requires a substantially larger sample size [4]. The choice between 80% and 90% power directly impacts the duration of the experiment and the traffic allocation required.[4]

The third dial is the significance level, commonly known as alpha, which controls the false positive rate, or Type I errors. Alpha is the statistical evidence required before declaring a result significant, typically set at 5% [2]. This means the experimenter accepts a 5% risk of detecting an effect that is entirely due to random variance.[2]

Tightening alpha from 5% to 1% reduces false positives but simultaneously reduces statistical power unless the experimenter compensates by adding significantly more data [4]. These three dials—MDE, power, and alpha—form a four-way mathematical constraint with sample size; fixing any three automatically determines the fourth [1].[1][4]

MetricGate provides a concrete 2026 example of this constraint in action: "Suppose your baseline conversion rate is 5%, you have 10,000 users per arm, alpha = 0.05, power = 0.80. Using the normal approximation for proportions, the MDE is about 0.6 percentage points" [1].[1]

In that scenario, the test can reliably detect a shift from 5.0% to 5.6%. If the smallest improvement that would justify launching the new feature is 0.3 percentage points, the experiment as designed simply cannot detect it [1]. The math forbids it, regardless of how long the dashboard runs.[1]

To resolve this mathematical gridlock, teams must either collect more users, run the test longer, switch to a more sensitive target metric, or reduce variance through advanced statistical techniques like CUPED [1]. Sample size remains the unavoidable toll paid to navigate the trade-offs between the risk of missing a winner and the operational cost of waiting for data.[1]