Why a 9% Response Rate Doesn't Mean a Poll is 91% Wrong

In 1997, the Pew Research Center successfully interviewed 36% of the households it called for its standard telephone surveys. By 2012, that figure had fallen to 9%, and it has hovered in the single digits ever since.[1][2]

To a lay observer, a poll that fails to reach 91% of its targets is a poll that is 91% wrong. The assumption is intuitive: if only a tiny fraction of the public is willing to answer a questionnaire, the resulting data must reflect a fringe minority rather than the general population.[4]

But in survey science, the relationship between how many people answer the phone and how accurate the resulting data is turns out to be remarkably weak. The metric that actually determines a poll's validity is not the response rate, but nonresponse bias.

Nonresponse bias occurs only when the people who choose not to participate in a survey differ systematically from those who do, specifically on the topics being measured. If the 9% who answer the phone hold the exact same distribution of views as the 91% who ignore it, the poll will be perfectly accurate despite the low yield.

"The potential for what pollsters call 'nonresponse bias' ... certainly is greater when response rates are low," wrote Scott Keeter, director of survey research at Pew Research Center. "But the mere existence of low response rates doesn't tell us anything about whether or not nonresponse bias exists."

Decades of methodological research back this up. A 2018 study by Richard Hendra and Aaron Hill analyzed extensive survey data and found no significant relationship between survey response rates and nonresponse bias.

To prove this in practice, Pew Research Center regularly benchmarks its 9% response rate telephone polls against high-cost, high-response-rate federal surveys like the General Social Survey, which often achieve response rates above 60%.[1]

In 2016, Pew compared its polling on 13 demographic, lifestyle, and health questions against these government benchmarks. The average absolute difference between the low-response telephone estimates and the high-response federal estimates was just 2.7 percentage points.[1]

This accuracy is not an accident; it is the result of post-fielding statistical adjustments. When pollsters collect their raw data, they know certain groups—often younger adults, lower-income individuals, and minorities—are less likely to respond.

To correct this, researchers apply demographic weighting. If census data shows that 18-to-29-year-olds make up 20% of the population, but they only make up 10% of the survey respondents, the answers from the young adults who did participate are mathematically weighted up to count for their true population share.

However, weighting only works if the young adults who answer the phone are politically and socially similar to the young adults who do not. This is where nonresponse bias can still creep in and break a poll, even after demographic adjustments.[4]

The most persistent vulnerability involves civic engagement. People who are willing to spend twenty minutes answering a pollster's questions are inherently more civically engaged than those who hang up.[1]

Because of this, surveys consistently overestimate behaviors like volunteering, attending community meetings, or voting in local elections. In a 2016 analysis of online panels, surveys overestimated the share of adults who worked with neighbors to fix a community problem by an average of 20 percentage points.[3]

When a survey tries to measure a trait that is directly correlated with the willingness to take surveys, nonresponse bias becomes a severe structural problem. But for most standard political, social, and economic questions, the bias remains minimal.

The polling industry's shift toward Address-Based Sampling and opt-in web panels is largely an attempt to control the soaring costs of finding that 9% of respondents, rather than a desperate bid to fix nonresponse bias.[2][4]

As long as pollsters can accurately model the electorate and weight their samples, a single-digit response rate is a financial headache for the survey firm, not a fatal flaw for the data. The true test of a poll is not how many people answered, but who was left out.