AI Models Hit a Wall on 'First Proof,' a Rigorous New Benchmark for Research Mathematics

Artificial intelligence has spent the last year conquering some of the most prestigious standardized tests on Earth, passing the bar exam, and even scoring gold medals at the International Mathematical Olympiad. These highly publicized milestones led to bold industry claims that AI systems were on the verge of autonomous scientific discovery. But when stripped of human assistance and forced to solve novel, unpublished research problems, the world's most advanced models hit a wall. On June 10, 2026, a coalition of elite mathematicians released the results of the "First Proof" benchmark, revealing a massive gap between AI pattern-matching and genuine mathematical creativity.[1][3]

The First Proof initiative was designed by eleven leading mathematicians from institutions including Harvard, Stanford, Yale, and UC Berkeley to solve a pervasive problem in AI evaluation known as "data contamination." Because large language models are trained on vast, opaque swaths of the internet, it is nearly impossible to tell if an AI is actually reasoning through a complex problem or simply regurgitating a solution it memorized from an obscure 1980s journal. Standard benchmarks have become increasingly unreliable as models inadvertently ingest the test questions during their training phases.[2][3]

To neutralize this memorization loophole, the First Proof team sourced ten highly specific mathematical problems directly from the unpublished, ongoing research of human mathematicians. Because these problems had never appeared online, in any preprint server, or in any public talk, the AI models were forced to reason entirely from scratch. The problems spanned diverse and highly specialized fields, including symplectic geometry, spectral graph theory, stochastic analysis, and algebraic combinatorics.[3][4]

The benchmark specifically focused on proving "lemmas"—intermediate propositions that mathematicians use as stepping stones to build larger, more complex theorems. Lemmas represent the actual daily labor of a working research mathematician. They are not artificially constructed puzzles designed for automatic grading, nor do they have clean numerical answers like competition math; they require building rigorous, end-to-end logical arguments that can survive peer review.[3][4]

The initiative's initial "First Batch" experiment, conducted in February 2026, highlighted exactly why a rigorous benchmark was necessary. During that informal round, major tech companies including OpenAI and Google DeepMind submitted solutions that appeared highly impressive at first glance. However, subsequent analysis revealed that these successes relied heavily on human intervention, a collaborative process researchers refer to as "Centaur Math."[3][4]

In Centaur Math, human experts steer the AI system, correcting its logical gaps, suggesting alternative approaches when it gets stuck, and manually selecting the best output from dozens of generated attempts. While this proves that AI is a highly effective interactive tool, it completely obscures the model's ability to generate a novel mathematical idea autonomously. The First Proof organizers realized that to measure true machine intelligence, they had to sever the human lifeline.[3][4]

The "Second Batch" of the First Proof benchmark, conducted in late May and graded in early June 2026, imposed draconian restrictions on the AI systems to test pure autonomy. The formal benchmark banned human steering entirely. Models were given exactly one shot to generate a proof via an API, with a strict 24-hour time limit and zero human intervention allowed during the generation process.[2][3]

The AI-generated proofs were anonymized, assigned animal code names like "Ocelot," "Badger," and "Marmot," and subjected to blind peer review. On June 4 and 5, human experts gathered at Harvard University's Center of Mathematical Sciences and Applications (CMSA) to grade the submissions. The referees applied the exact same rigorous standards used by academic mathematics journals, evaluating the proofs for logical soundness, clarity, and the presence of a core conceptual breakthrough.[2][3]

The results were sobering for AI advocates. According to Nature, human mathematicians significantly outperformed the AI models across the board. Scientific American characterized the collective AI performance as a "C-," noting that the systems struggled profoundly when forced to formulate the creative leaps required for research-level math without a human guide.[1]

Without human steering, the models frequently produced what reviewers called "confident but flawed" proofs. The AI systems proved highly adept at generating pages of plausible-sounding mathematical notation, perfectly mimicking the tone and structure of an academic paper. However, beneath the polished formatting, the proofs frequently contained fatal logical errors, circular reasoning, or fundamental misunderstandings of the core geometric or algebraic concepts.[1]

The benchmark highlighted a specific vulnerability in current AI architectures: the inability to recognize their own boundaries and simply say, "I don't know." In mathematics, where a single incorrect assumption or hallucinated logical bridge invalidates an entire proof, this tendency to confidently guess is a critical liability. The models lacked the self-correction mechanisms that human mathematicians use to realize when a line of inquiry has hit a dead end.

Despite the low overall scores, the First Proof results were not entirely negative. The models did succeed at executing routine algebraic manipulations, verifying known steps, and translating mathematical concepts between different formats. This suggests that their immediate utility lies in acting as high-speed calculators and verification engines, freeing human mathematicians from tedious symbolic manipulation so they can focus on higher-level theory.[1][4]

The organizers drew a philosophical distinction between computation and conceptual discovery, quoting conceptual artist Sol LeWitt: "the idea becomes a machine that makes the art." The AI models proved excellent at operating the machine—executing the mechanical steps of a proof once a path was chosen—but they consistently failed to generate the initial "idea" or conceptual leap required to solve a novel problem.[3]

Rather than dismissing AI, the First Proof organizers view these results as a vital, uplifting recalibration for the scientific community. By establishing an objective, contamination-proof baseline, mathematicians now have a clear, transparent picture of where AI excels and where human intuition remains irreplaceable. The consensus is clear: while AI will undoubtedly accelerate the pace of mathematical research, the spark of genuine discovery remains a distinctly human domain.[1][2][3]