Observed Power Collapses to a Deterministic Function of the P-Value: Why Post-Hoc Power Calculations Cannot Explain Null Findings

In June 2019, statisticians demanded the retraction of a paper published by Harvard researchers, citing the use of a "nonsense statistic" to justify their conclusions. The dispute centered on a mathematical practice that remains pervasive in scientific literature: calculating statistical power after a study has already finished.[1]

This practice, known as post-hoc or observed power analysis, is frequently used by researchers to explain away a null result. When an experiment fails to find a significant effect, authors often calculate the power using their observed data to argue the study was simply too small.[4][5]

The mathematical reality, however, is that observed power provides absolutely no new information about the experiment. It is not an independent measure of a study's quality, but rather a direct, deterministic mathematical transformation of the p-value that the study already produced.[4][6]

"The pervasive fallacy of power calculations for data analysis" has been documented for decades, yet it continues to slip past peer review. Because the calculation uses the exact same variance and effect size that generated the p-value, the two metrics are inextricably locked together.[4]

If a researcher knows their p-value, they already know their post-hoc power, making the secondary calculation entirely redundant. The illusion of gaining new insight from this metric actively misleads scientists about the true uncertainty in their data.[2][5]

The Mathematics of the Collapse

To understand why this collapse occurs, one must look at how statistical power is defined before an experiment begins. A priori power is the probability of detecting a specific, pre-defined effect size, assuming that this effect truly exists in the population.[2][4]

Before collecting data, researchers estimate the expected variance and choose a minimum clinically important difference to calculate how many subjects they need. This forward-looking calculation is entirely valid and forms the bedrock of rigorous experimental design across all scientific disciplines.[5][6]

The fatal error occurs when researchers substitute their observed effect size into the power equation after the data is collected. By replacing the theoretical effect size with the exact difference found in the sample, the formula collapses into a restatement of the test statistic.[4]

John Hoenig and Dennis Heisey demonstrated in 2001 that for any standard statistical test, observed power is a strictly decreasing function of the p-value. There is a one-to-one mapping between the two numbers, meaning one cannot change without the other changing in exact proportion.[4]

This mathematical lock creates a rigid boundary: if a study's p-value is exactly 0.05, the post-hoc power calculated from that data will always be exactly 50 percent. It is a mathematical certainty, regardless of the sample size, the variance, or the field of study.[4][6]

Consequently, any study that fails to reach statistical significance—producing a p-value greater than 0.05—will mathematically guarantee a post-hoc power of less than 50 percent. The researcher will always find that their non-significant study was "underpowered" to detect the effect they observed.[2][4]

The Circular Logic of Excusing Null Results

This deterministic relationship creates a dangerous loop of circular logic in scientific publishing. A researcher runs a trial, finds a p-value of 0.20, and then calculates an observed power of 15 percent, which they present as a separate finding.[5]

They then argue in their discussion section that the lack of statistical significance should not be taken as evidence that the treatment failed. Instead, they claim the 15 percent power proves the study was simply ill-equipped to find the truth, keeping the hypothesis alive.[4][5]

As researchers writing in The BMJ noted in 2002, "Power is indeed irrelevant in interpreting completed studies." Using low observed power to excuse a high p-value is equivalent to saying that the p-value is high simply because the p-value is high.[3]

The argument implies that if the sample size had been larger, the result would have been significant. However, this assumes the observed effect size would remain perfectly constant as more data was collected, which contradicts the fundamental nature of sampling variance.[2][6]

If the true effect in the population is actually zero, collecting more data will simply drive the observed effect size closer to zero. A larger sample size in that scenario would not produce a significant p-value, exposing the flaw in the post-hoc justification.[4][5]

Confidence Intervals as the Solution

The persistence of post-hoc power calculations stems from a genuine need to interpret non-significant results. Researchers and clinicians need to know whether a null finding means the treatment truly does not work, or if the data is simply too noisy to tell.[2][5]

Statistical methodologists have long advocated for a superior alternative that actually answers this question: the confidence interval. Unlike observed power, a confidence interval provides a range of plausible values for the true effect, grounded in the data actually collected.[2][4]

If a confidence interval is very wide and includes both zero and large, clinically meaningful effects, the study is genuinely inconclusive. The data simply does not contain enough information to rule out either a massive benefit or a complete lack of efficacy.[2][6]

Conversely, if the confidence interval is narrow and tightly clustered around zero, the study provides strong evidence that no meaningful effect exists. This allows researchers to confidently declare a true negative finding, rather than leaving the question perpetually open.[2][5]

By focusing on the precision of the estimate rather than a redundant transformation of the p-value, confidence intervals separate the size of the effect from its statistical significance. They reveal exactly what the data can and cannot support.[2][4]

Reforming Editorial Standards

Despite decades of mathematical proofs demonstrating its invalidity, post-hoc power continues to be requested by peer reviewers and published in major journals. The intuitive appeal of the metric often overrides the statistical reality of its circular nature.[1][5]

The 2019 Harvard paper dispute highlighted by Retraction Watch underscores the real-world consequences of this fallacy. When flawed statistical logic is used to salvage null findings, it can misdirect future research funding and obscure the true efficacy of medical interventions.[1]

Furthermore, the reliance on observed power actively harms the scientific record by preventing the publication of definitive negative trials. When a well-designed study finds no effect, that information is crucial for guiding future research and protecting patients from ineffective treatments.[2][5]

If authors successfully argue that their study was simply underpowered based on a post-hoc calculation, the intervention is treated as though it might still work. This encourages other teams to waste resources repeating the same failed experiments in hopes of finding a different result.[4][6]

The definitive solution lies in rigorous a priori planning before a single data point is collected. By defining the minimum clinically important difference and powering the study to find it, researchers ensure their results will be informative regardless of the final p-value.[5][6]

Eradicating this practice requires a shift in how scientific journals evaluate negative results. Editors must enforce policies that ban post-hoc power calculations based on observed effect sizes, directing authors to report confidence intervals instead.[3][6]

The mathematical truth remains absolute: once the data is collected and the p-value is known, the statistical power of that specific sample is a closed loop. True scientific discovery relies on acknowledging what the data actually says, rather than calculating excuses for what it did not.[4][5]