The Counterfactual Problem: Why Observational Data Can Never Prove Causation Without a Causal Model

On March 25, 2015, researchers at the Federal Reserve Bank of Cleveland published Working Paper 15-05, a dense mathematical treatise that formalized a quiet revolution in how economists understand reality. The paper did not announce a new interest rate policy or forecast inflation. Instead, it drew a hard mathematical boundary between two competing frameworks for understanding cause and effect: the Rubin Causal Model and Structural Causal Models. The distinction matters profoundly because modern society is drowning in observational data, yet data alone—no matter how vast—cannot mathematically prove what causes what.[1]

The core argument is transparent: we cannot simply look at a spreadsheet of historical events and deduce causality. As the National Institutes of Health's repository notes, the entire field of causal inference is built on the concept of counterfactuals. A counterfactual asks what would have happened to a specific subject if an intervention had not occurred. Because we cannot observe both realities simultaneously, causal inference is fundamentally a missing data problem.[3]

Historically, the dominant approach to this missing data was the Rubin Causal Model, named after statistician Donald Rubin. This framework relies on "potential outcomes" and assumes that if we control for enough variables, the treatment assignment becomes effectively random—a condition known as "conditional exchangeability." It is a powerful tool for estimating effects when the intervention is clearly defined, such as a medical treatment, and the variables are strictly controlled.[1]

However, the strongest counter-argument to the Rubin model emerged from computer science, spearheaded by Judea Pearl. In his 2009 foundational text, Causality: Models, Reasoning, and Inference, Pearl demonstrated that the potential outcomes framework lacks the mathematical vocabulary to express the structural relationships between variables before the data is collected. Pearl argued that without a structural model, statisticians are flying blind, unable to verify if their statistical controls are actually helping or hurting.[5]

Pearl introduced Directed Acyclic Graphs (DAGs)—simple visual maps of arrows connecting variables—to explicitly state causal assumptions. These structural models prove that simply controlling for more variables does not automatically bring you closer to the truth. In fact, controlling for a specific type of variable called a "collider" mathematically introduces bias where none existed, creating a false correlation between two entirely independent causes.[5]

This is where the "Big Data" empiricist view falls short. Industry analysts frequently ask whether observational studies can show cause and effect, especially with the advent of machine learning. The algorithmic instinct is to feed every available variable into a model and let the computer sort it out. But algorithms operate on the lowest rung of what Pearl calls the "Ladder of Causation"—association. They excel at seeing that symptom A is correlated with disease B, but they cannot deduce whether A causes B, B causes A, or a hidden factor C causes both.[4][5]

To move up the ladder to intervention (what happens if we change A?) and counterfactuals (what would have happened if we had not changed A?), we must supply the machine with a causal model. The data cannot generate the model; the model must constrain the data. As the Federal Reserve researchers highlighted, structural models define the causal mechanisms explicitly, allowing researchers to test whether their assumptions are compatible with the observed data.[1]

The medical field has increasingly recognized this hard limit. A comprehensive review published in the Journal of Epidemiology and Community Health emphasizes that effect estimation using observational data requires rigorous causal inference frameworks. Epidemiologists can no longer simply run regressions and append a disclaimer that "correlation does not imply causation." They must explicitly state their causal assumptions using DAGs before calculating a single p-value.[2]

The transparency of this approach is its greatest strength. When researchers publish a structural causal model, they are putting their assumptions on the table. If a critic disagrees with the conclusion, they do not have to argue vaguely about "omitted variable bias." They can point to the exact arrow in the DAG that they believe is missing or drawn backward, turning a philosophical debate into a testable mathematical proposition.[5]

This dialectic between the Rubin and Pearl frameworks has ultimately strengthened both. The Rubin model provides the rigorous statistical machinery for estimating the size of a causal effect, while Pearl's structural models provide the logical blueprint required to ensure that the estimation is mathematically valid in the first place. Together, they form the bedrock of modern causal inference.[1][5]

The implications extend far beyond academic statistics. From tech companies running A/B tests to governments evaluating policy interventions, the reliance on observational data is absolute. Yet, as the literature confirms, without a structural model, every causal claim derived from observational data remains mathematically unproven, vulnerable to hidden confounders and collider bias.[3]

The next frontier is integrating these causal models directly into artificial intelligence. Until AI systems can process counterfactuals—reasoning about what would have happened under different circumstances—they will remain sophisticated pattern matchers rather than true reasoning engines. The mathematical floor has been established: data is the evidence, but the causal model is the logic that makes the evidence meaningful.[6]