Independent Testers Find 37-Point Gap in GPT-6 Astra's Key Intelligence Benchmark

For an artificial intelligence benchmark to measure raw capability, the testing environment—the software harness wrapped around the model—must not provide memory or reasoning shortcuts that the model cannot generate itself. In the case of OpenAI's newly released GPT-6 Astra, that condition does not hold uniformly, and the difference in testing environments is worth 37 percentage points on a flagship evaluation.[1][6]

When OpenAI launched GPT-6 Astra on September 3, 2026, the company positioned it as a generational leap in reasoning and agentic work. The headline claim rested heavily on a near-perfect 99.9 percent score on ARC-AGI-3, a notoriously difficult test of abstract puzzle-solving that requires an AI to infer rules in novel environments.[2][6]

However, independent testing by the ARC Prize Foundation revealed that this near-perfect score depends entirely on how the model is allowed to remember its past steps. When tested under the standard, provider-neutral harness—which forces the model to rely only on the notes it explicitly chooses to write down—GPT-6 Astra scored 62.7 percent.[1][6]

The 99.9 percent result was achieved using a custom "Provider Adapter" harness. This specialized environment preserves the model's opaque reasoning states between requests and uses background compaction to reuse prior computational work, effectively giving the AI a continuous, hidden memory loop that bypasses the need to re-derive answers from scratch.[1]

This 37-point gap is not a flaw in the model, but a measurement of how much Astra relies on retaining its own hidden reasoning across turns. It highlights a growing reality in artificial intelligence: the scaffolding around a model is now as critical to its performance as the neural network itself.[1][6]

The discrepancy became part of a broader conversation about benchmark transparency following the model's release. Within days of the launch, OpenAI quietly revised several published figures on its site, including halving Astra's reported hallucination rate from 4.2 percent to 2.0 percent before reverting it back toward the original number.[3][5]

Addressing the shifting numbers, an OpenAI spokesperson stated, "We always verify evals before publication so adjustments between draft and final version are normal." Yet the revisions underscore how sensitive headline metrics are to minor configuration changes.[5]

Independent evaluators have since placed Astra's raw intelligence in context. Artificial Analysis, which runs a standardized Intelligence Index across the industry, scored GPT-6 Astra at 61.2 on September 9, 2026. This places the new flagship effectively tied with its predecessor, GPT-5.6 Sol, which scored 60.9, and slightly behind Anthropic's Claude Fable 5.1 at 65.7.[4]

Where Astra undeniably excels is in agentic efficiency and computer use. On the Artificial Analysis Coding Agent Index, Astra matches Fable 5.1 while operating at roughly 60 percent of the cost per task, using significantly fewer output tokens to reach the same conclusions.[4]

The model also beat the human action-efficiency baseline in the ARC-AGI-3 environment, completing 96.0 percent of the test levels using fewer actions than the median human participant.[1]

The emergence of this 37-point gap marks a shift in how AI capabilities are evaluated. As models become capable of long-horizon tasks, the distinction between a model's raw weights and its surrounding software architecture is blurring, leaving independent auditors to define where the algorithm ends and the tool begins.[1][4][6]