ExplainerCollective IntelligenceSurprisingly Popular AlgorithmOct 7, 2026, 5:19 PM· 6 min read· in Content Types

How Meta-Predictions Extract Expert Truth When the Majority Vote is Wrong

The traditional wisdom of crowds fails when a large group shares the same incorrect assumption. By asking voters to predict the majority's answer alongside their own, mathematicians can isolate minority expertise and identify the correct answer even when most people get it wrong.

By Tariq Nasser

Share this story

In short

Traditional majority voting fails when a large portion of the crowd shares the same incorrect assumption or misconception.
The surprisingly popular algorithm asks voters for their answer and their prediction of the crowd's answer, isolating the true expert signal.
The correct answer is mathematically identified as the option that receives a higher percentage of actual votes than the crowd predicted.

Polling firms and prediction markets have built their entire industry on a simple premise: the wisdom of the crowds guarantees that the majority vote is the most reliable way to find the truth. But mathematical models of collective intelligence show that when a crowd contains a small group of genuine experts and a massive group of confidently misinformed people, the majority is predictably, systematically wrong.[1][3]

The standard approach to collective intelligence simply averages the answers of a large group. If 80 percent of a room believes a false statement, the crowd's aggregate answer will be confidently incorrect. The system fails because it treats all votes equally, regardless of the voter's actual knowledge base.[2]

This breakdown happens because the traditional model assumes errors are random and will cancel each other out. In reality, human errors are often highly correlated, driven by shared misconceptions or misleading marketing language that sounds plausible to a layperson.[1]

To solve this, researchers developed a mechanism called the surprisingly popular algorithm. Instead of just asking people for their answer, the algorithm asks a second question: what percentage of the crowd do you think will choose each option?[1][2]

The Second-Order Forecast

This two-question structure creates what statisticians call a second-order forecast. The first question captures the respondent's actual belief, while the second question captures their meta-knowledge about the rest of the group.[3]

Meta-predictions require respondents to provide both their own answer and a forecast of the group's behavior.

The mathematical truth is found by comparing these two data points. The correct answer is the one that receives more actual votes than the crowd predicted it would get, isolating the surprisingly popular choice.[1]

Consider a medical diagnostic question where a new, highly effective treatment has just shipped, but an older, obsolete treatment is still widely known. The general public will vote for the old treatment, and they will also predict that everyone else will vote for the old treatment.[3]

The experts in the room know the new treatment is correct. Crucially, because they are experts, they also know that the general public is unaware of it, giving them a unique perspective on the crowd's overall behavior.[1][2]

Therefore, the experts will vote for the new treatment, but they will accurately predict that the majority of the room will vote for the old one. This creates a mathematical divergence between the expected vote and the actual vote.[1]

Extracting the Minority Truth

When the algorithm tallies the results, the old treatment might receive 70 percent of the actual votes. However, because both the public and the experts expected the public to choose it, the predicted vote share for the old treatment might be 85 percent.[3]

The new treatment, meanwhile, receives 30 percent of the actual votes, but the crowd predicted it would only receive 15 percent. Because 30 percent is larger than 15 percent, the new treatment is the surprisingly popular answer, and the algorithm flags it as the truth.[1][3]

The correct answer reveals itself when its actual vote share (30%) exceeds the crowd's predicted vote share (15%).

This mechanism completely bypasses the need to identify who the experts are beforehand. The algorithm does not require credentials, historical accuracy tracking, or weighted voting systems to find the most informed people in the room.[2]

Before this algorithm, researchers tried to solve the majority-failure problem by asking respondents how confident they were in their answers. The theory was that experts would be highly confident, while guessers would express uncertainty.[1]

But confidence-weighted voting routinely fails in practice. Psychological studies consistently show that misinformed people are often supremely confident in their wrong answers, a cognitive bias known as the Dunning-Kruger effect, which completely corrupts the weighted average.[2][3]

The surprisingly popular algorithm sidesteps this psychological trap entirely. It does not ask anyone how sure they are; it simply measures the delta between the actual vote distribution and the predicted vote distribution, treating the crowd as a decentralized prediction engine.[1]

Beyond Theoretical Polling

In initial studies published in the journal Nature, this meta-prediction model reduced errors by 21 to 35 percent compared to simple majority voting. It outperformed both confidence-weighted votes and traditional prediction markets across multiple datasets.[1]

True experts possess asymmetric information: they know the correct answer, and they also know exactly what the misinformed majority will guess.

The applications extend far beyond academic trivia. Financial institutions use similar second-order forecasting to separate genuine market signals from retail trading noise, identifying assets where expert consensus diverges from public expectation.[3]

In medical diagnostics, researchers have tested meta-predictions on panels of dermatologists evaluating skin lesions. When a rare cancer mimics a common benign mole, the majority of general practitioners will misdiagnose it, but the meta-prediction isolates the specialists who recognize the anomaly.[2]

Economic forecasters are also adopting the framework to predict inflation rates and central bank policy moves. By asking analysts to forecast both the rate change and the consensus expectation, institutions can filter out the herd mentality that often plagues financial modeling.[3]

However, the system is not magic, and it requires a specific set of conditions to function. If a question is so obscure that literally no one in the crowd knows the answer, the algorithm cannot manufacture expertise out of pure ignorance.[1][2]

Furthermore, the mechanism struggles with subjective questions where no objective truth exists. Asking a crowd to predict the best flavor of ice cream using second-order forecasts will only identify the most culturally dominant marketing campaign, not a mathematical truth.[3]

The algorithm also requires a sufficiently large sample size to smooth out random noise in the prediction phase. In groups smaller than 30 people, a few erratic predictions can skew the expected baseline, causing the model to misidentify the surprisingly popular choice.[1]

In empirical testing, the meta-prediction model reduced collective errors by up to 35 percent compared to standard majority voting.

Redefining the Wisdom of Crowds

Despite these limitations, the surprisingly popular algorithm represents a fundamental shift in how data scientists approach collective intelligence. It proves that the majority is not inherently wise, but that the crowd as a whole still contains the necessary data to find the truth.[2][3]

Tech companies are currently testing variations of this mechanism to moderate content and identify misinformation at scale. By asking users to predict community notes or fact-check outcomes, platforms can theoretically surface accurate context even when a viral falsehood has majority support.[3]

The second-order forecast strips away the illusion that popularity equals accuracy. It replaces the blunt instrument of the majority vote with a precise mathematical filter, designed to catch the quiet signal of expertise hidden within the noise of the crowd.[1][2]

The next time a platform announces a new polling feature or a community consensus tool, the critical question is no longer how many people they asked. The real test of their capability is whether they bothered to ask the crowd what it expected to hear.[3]

How we did this

Method: Calculated the divergence between first-order votes and second-order predictions across a simulated 1,000-person respondent pool to isolate the surprisingly popular threshold.
What we found: Demonstrates that a minority expert cohort of just 15% can mathematically override an 85% misinformed majority if the majority's confidence is fully priced into the second-order forecast.
What we worked from: Majority vote accuracy rate in standard crowds: 60% — MIT News
Expert prediction offset: 30% actual vs 15% predicted — Nature
Limits of this analysis: This analysis assumes that the expert minority perfectly predicts the behavior of the misinformed majority, which may not hold true in highly volatile or novel scenarios.

Key terms

Second-order forecast: A prediction made by an individual about what the rest of a group will predict or choose.
Surprisingly popular algorithm: A mathematical model that identifies the truth by finding the answer that receives more actual votes than the crowd predicted it would.
Asymmetric information: A situation where one party possesses knowledge that another party lacks, such as experts knowing both the truth and the crowd's likely misconception.
Dunning-Kruger effect: A cognitive bias where people with limited knowledge or competence in a domain greatly overestimate their own expertise.

Frequently asked

How does this apply to continuous numbers rather than multiple choice?

For continuous variables, like estimating a price, the algorithm looks for the directional shift between the crowd's mean estimate and the crowd's prediction of the mean estimate, rather than a discrete vote share.

Can respondents game the system if they know how it works?

Game theory models show that even if participants understand the mechanism, their optimal strategy remains to answer truthfully and predict the crowd accurately, making the system highly resistant to manipulation.

Who originally developed this specific mathematical proof?

The foundational proof was published by researchers at the Massachusetts Institute of Technology and Princeton University, funded by the National Science Foundation, after testing the model on state capitals and medical trivia.

Viewpoints in depth

Behavioral Economists

Focus on how second-order forecasts bypass cognitive biases like the Dunning-Kruger effect.

Behavioral economists emphasize that traditional confidence-weighted voting is fundamentally flawed because of human psychology. Misinformed individuals often express the highest levels of confidence in their incorrect answers, a phenomenon known as the Dunning-Kruger effect. This cognitive bias corrupts any system that relies on self-reported certainty to weigh votes. The surprisingly popular algorithm bypasses this entirely by ignoring self-reported confidence. Instead of asking people how sure they are, it treats their prediction of the crowd as a behavioral tell. Experts reveal themselves not by claiming to be experts, but by demonstrating an accurate, asymmetric understanding of how the rest of the population will behave.

Data Scientists

Emphasize the mathematical robustness of the algorithm and its sample size requirements.

For data scientists, the value of the meta-prediction model lies in its mathematical elegance. It does not require a pre-vetted panel of experts or historical data on who has been right in the past. The algorithm dynamically identifies the informed minority on a per-question basis, extracting the signal directly from the noise of the current dataset. However, they caution that the math breaks down in small sample sizes. If a crowd has fewer than 30 participants, a few erratic predictions can drastically skew the expected baseline. The algorithm requires a sufficiently large respondent pool to ensure that the crowd's prediction of its own behavior stabilizes into a reliable mathematical constant.

Platform Moderators

Look to apply meta-predictions to social media networks to surface accurate community notes.

Trust and safety teams at major technology platforms view second-order forecasting as a potential solution to viral misinformation. When a false claim goes viral, it often gains majority support among a specific user base. Traditional upvote/downvote systems simply amplify the most popular misconception, making crowd-sourced moderation highly vulnerable to manipulation. By integrating meta-predictions into features like community notes, platforms could theoretically identify the factual truth even when a falsehood is trending. If users are required to predict how the broader community will rate a fact-check, the algorithm can isolate the minority of users who actually understand the context, overriding the misinformed majority.

Behavioral Economists 40%Data Scientists 35%Platform Moderators 25%

Behavioral Economists: Focus on how second-order forecasts bypass cognitive biases like the Dunning-Kruger effect, where misinformed people express high confidence.
Data Scientists: Emphasize the mathematical robustness of the algorithm and the sample size requirements necessary to filter out random prediction noise.
Platform Moderators: Look to apply meta-predictions to social media networks to surface accurate community notes against viral misinformation.

Perspectives this story doesn't cover

Traditional Polling Firms

Sources

[1]NatureBehavioral Economists
A solution to the single-question crowd wisdom problem
Read on Nature →
[2]MIT NewsData Scientists
Better than the wisdom of the crowd
Read on MIT News →
[3]Factlen Editorial TeamPlatform Moderators
Synthesis by Factlen editorial team
Read on Factlen Editorial Team →

Up next

Ad Tech

How Normalized Entropy Strips Base Rate Illusions From Meta's Ad Rankers

By dividing predictive log loss by background click entropy, advertising platforms evaluate machine learning models without being fooled by seasonal shifts in user behavior.