New AI Optimization Framework Beats Claude Code and Codex by 2.5x on Same Compute Budget

The promise of autonomous AI agents has always been their ability to work tirelessly, but engineering teams are increasingly discovering a frustrating reality: a loop is not the same as progress. When tasked with optimizing a complex system, standard coding agents like Claude Code or Codex often fall into a tedious cycle of trial-and-error. They might tweak chunking strategies, adjust retrieval methods, and rewrite system prompts all at once, creating an entangled mess where it becomes impossible to attribute which specific change actually improved the system.[1]

This limitation stems from how current agent architectures handle state and memory. They treat each attempt in isolation, relying on a scrolling context buffer that eventually loses the thread of what was tried, what failed, and why. As a result, these systems frequently repeat the same mistakes or produce "improvements" that look good in development but fail spectacularly when deployed in production environments.[1][2]

To address this bottleneck, researchers from Renmin University of China and Microsoft Research have introduced Arbor, a new framework designed to upgrade AI-driven optimization from a sequence of isolated guesses into a cumulative learning process. The system fundamentally changes how autonomous research is conducted by introducing a persistent data structure that tracks the history of experiments, ensuring that the AI actually learns from its past iterations.[1][2]

The results of this architectural shift are striking. In practical evaluations across real-world engineering tasks, Arbor delivered more than 2.5 times the verifiable performance gains of standard AI coding agents, all while operating under the exact same resource and compute budget. This efficiency leap suggests that the primary bottleneck in autonomous optimization is not a lack of raw intelligence or compute power, but rather a lack of structured scientific methodology.[1][2][5]

At the core of Arbor's success is a mechanism the researchers call "Hypothesis-Tree Refinement." Instead of allowing a single agent to directly mutate a target codebase in one pass, Arbor separates the strategic direction of the research from the ground-level execution. It organizes hypotheses, experimental evidence, and distilled insights into a branching tree structure, creating an auditable trail of the entire optimization process.[2]

This separation of concerns is implemented through two distinct roles: a long-lived "coordinator" and multiple short-lived "executors." The coordinator acts like a principal investigator in a research lab; it never directly edits the code. Instead, it observes the accumulated evidence, generates new hypotheses, and decides which branches of the tree are worth exploring further based on empirical results.[1][4]

When the coordinator identifies a promising direction, it dispatches an executor to test that specific hypothesis in a completely isolated worktree. This isolation is critical for enterprise applications, such as optimizing a Retrieval-Augmented Generation (RAG) pipeline. By testing one lever at a time—rather than changing the prompt, the chunking, and the retrieval method simultaneously—Arbor ensures that every performance gain can be accurately attributed to a specific intervention.[1][2]

The empirical evidence supporting Arbor's approach is robust across multiple industry benchmarks. On the MLE-Bench Lite machine learning engineering benchmark, Arbor, when equipped with the GPT-5.5 backbone model, achieved an 86.36% "Any Medal" rate, marking the strongest result among all benchmarked systems and significantly outpacing traditional coding agents.[2][4]

The framework's superiority becomes even more apparent in complex, multi-step optimization tasks. On the BrowseComp task, which requires the AI to optimize a search agent, Arbor successfully improved the system's held-out accuracy from a baseline of 45.33% to 67.67%. In contrast, traditional single-trajectory agents like Codex and Claude Code stalled at 50% and 53.33%, respectively, unable to navigate the long-horizon search space effectively.[1][2]

Beyond raw performance, Arbor demonstrates a crucial resilience against overfitting—a common pitfall where an AI optimizes perfectly for the development test but fails on unseen data. During the Terminal-Bench 2.0 task experiments, Claude Code achieved a high development score of 75, but its performance dropped to 71 on the held-out data, indicating that it had simply memorized the test parameters.[1][2]

Arbor, conversely, exhibited a more generalized learning pattern. It recorded a lower development score of 72.22 but achieved the highest held-out score of 77.36. This indicates that the Hypothesis-Tree Refinement process successfully filters out hacky, brittle solutions, ensuring that the resulting optimizations actually transfer to real-world applications where conditions are less predictable.[1]

The framework also proved capable of cross-task transfer. After optimizing the search harness for the BrowseComp task, the researchers took Arbor's optimized codebase and tested it on two entirely unrelated search-agent tasks. The system maintained its high performance, demonstrating that the insights accumulated by the coordinator were fundamentally sound rather than hyper-specialized to the original training environment.[1]

For enterprise AI teams, Arbor represents a shift from manually prompting coding agents to orchestrating automated research loops. The framework is available as an open-source research system and includes an Agent Skill Suite that can be loaded inside existing tools like Codex and Claude Code, allowing developers to leverage Arbor's methodology within their current workflows without abandoning their preferred models.[1][2]

However, the researchers are transparent about the system's limitations and the specific conditions required for it to succeed. Arbor is not a universal solution for all coding tasks; it is explicitly not recommended for real-time latency optimization or obvious, one-line bug fixes where the overhead of managing a hypothesis tree would be counterproductive and slow down development.[3][4]

More importantly, Arbor's effectiveness is strictly bounded by the quality of the evaluation metric it is given. Because the system is so efficient at optimizing toward a target, a flawed or gameable metric will simply result in the AI reaching an untrustworthy result much faster, amplifying any human errors made during the initial setup phase.[1][5]

As Jiajie Jin, co-author of the paper, cautioned, if the goal is vague or the metric is easy to hack, long-running automation will produce "improvements" that nobody actually wants. This places a new burden on human engineers: as AI systems become better at autonomous optimization, the human role shifts from writing the code to rigorously defining the exact metrics of success.[1]

Ultimately, Arbor highlights a growing divergence in the landscape of AI agents. While some systems are optimizing for rapid, iterative code generation, frameworks like Arbor are pioneering the space of long-horizon hypothesis management. By proving that cumulative learning structures can yield a 2.5x performance multiplier without additional compute, Arbor sets a new standard for how autonomous research will be conducted in the future.[1][5]