Evidence Pack: The Accuracy of Synthetic Data in Replicating Human Survey Responses

Commercial vendors and artificial intelligence platforms claim that "synthetic respondents"—large language models programmed with demographic traits—can replace human survey panels, boasting 85% to 95% accuracy on quantitative trends. But a wave of 2026 empirical benchmarks sets the data directly against that pitch. When subjected to rigorous testing, silicon samples fail to replicate human variance, collapse under subgroup analysis, and exhibit severe sycophancy bias that renders them unsafe for high-stakes decisions.[7][8]

The intuition behind synthetic sampling relies on the vast latent social information embedded in large language models. Instead of recruiting 500 real people, a researcher prompts an artificial intelligence to simulate 500 distinct personas, feeding the model demographic structures and behavioral parameters. The system then predicts how those personas would answer a given survey. Proponents argue this neuro-symbolic approach predicts decisions rather than just generating text, eliminating fieldwork time and respondent fatigue.[8]

The empirical reality, however, diverges sharply from the sales pitch. In January 2026, the polling and analytics firm Verasight published a direct comparison between a nationally representative sample of 2,000 United States adults and a matched synthetic sample. Across topics spanning politics, healthcare, and consumer behavior, the synthetic respondents consistently failed to replicate human response patterns.[1]

The Verasight benchmark revealed an overall mean absolute error of 14.5 percentage points across all single-answer response options. While topline results occasionally approximated human data within a four-point margin, the illusion of accuracy vanished when researchers looked closer. At the demographic subgroup level, the error between human and AI-generated samples ballooned to 10 points on average, spiking to a 30-point divergence for the smallest population segments.[1]

A separate May 2026 academic study tested the assumption that giving the artificial intelligence more granular demographic data would improve its accuracy. Researchers fielded a survey to 1,917 human adults via the Prolific platform and generated three tiers of synthetic data, ranging from shallow aggregate predictions to "deep" specification that created 1,917 individual digital twins matching the exact demographics of the human panel.[5]

The results contradicted the core premise of digital twin technology. Increased specification did not improve accuracy. In fact, the deep specification with individual-level demographic matching paradoxically performed significantly worse than the shallow models. Furthermore, applying standard demographic post-stratification weights to the synthetic data failed to meaningfully approximate the weighted human estimates, proving that statistical corrections cannot salvage fundamentally flawed synthetic baselines.[5]

The failures extend beyond political polling into commercial market research. In July 2026, the insight and analytics group Strat7 compared a nationally representative sample of 3,000 real humans against surveys run by two synthetic data companies. The test included a Van Westendorp pricing exercise, which asks respondents to identify price points that feel too cheap, cheap, expensive, and too expensive.[2]

Purely synthetic respondents broke the logical order of the pricing exercise 68% of the time, fundamentally failing to understand the linear relationship of money. When blended with real data, that failure rate dropped to 32.8%. Even when the artificial intelligence managed to maintain logical consistency, the prices generated by synthetic respondents were generally 16% higher than those provided by actual consumers, introducing a dangerous premium bias into potential product strategies.[2]

These quantitative failures stem from structural limitations in how large language models operate. A 2026 review of synthetic-user experiments, alongside research from the Stanford Institute for Human-Centered Artificial Intelligence, documented two persistent flaws: sycophancy bias and majority-opinion convergence. Synthetic personas drift toward whatever answer the question framing implies the researcher wants to hear.[4]

Because artificial intelligence models lack lived experience, they converge on the mean of their training data. They cannot simulate the fatigue, distraction, or spontaneous edge cases that characterize real human behavior. Real participants surprise researchers by using products in unanticipated ways or bringing stories that reframe the core question. Synthetic respondents simply pattern-match, producing uniform, positive-skewed answers that lack emotional nuance.[4][7]

The research industry has recognized this gap between vendor promises and empirical reality. A May 2026 survey of 150 research professionals by User Interviews found that while 97% of researchers use artificial intelligence in some part of their workflow, only 8% trust AI-generated participants for decision-grade calls. The profession has adopted the technology comprehensively but explicitly rejected this specific application for final validation.[6]

Consequently, the mature 2026 consensus has shifted away from replacement and toward a two-phase hybrid stack. Survey platforms like Qualtrics now explicitly state that "synthetic data augments human research, it does not replace it." In this model, researchers use synthetic panels in the first phase to rapidly test hypotheses, screen early concepts, and identify flaws in survey wording before spending budget on fieldwork.[3][4]

Once the field of options is narrowed, the surviving concepts are validated against real human panels. Human respondents remain the source of truth that anchors the models; without continuous human data, synthetic systems would quickly lose their calibration to shifting cultural and behavioral realities. The dividing line in modern research is no longer whether to use artificial intelligence, but knowing exactly which questions require a human to answer them.[3]