A 100% score that OpenAI couldn’t touch
Nvidia dropped some research on Friday that flips a lot of assumptions about AI agents on their head. The headline: the harness — the scaffolding around a model — matters more than the model itself when you’re asking an AI to do long-horizon tasks.
The proof is stark. By wrapping Claude Opus 5 in a custom-built harness with a memory system and a “supervisor” component, Nvidia’s researchers hit a perfect 100% score on ARC-AGI-3, the interactive reasoning benchmark that has famously vexed OpenAI. Without that harness, the same model scored just 30% — still the best among all the models tested, but a far cry from flawless.
That gap is the whole story.
What exactly is a harness?
Most people think of an AI agent as the model underneath. Nvidia’s Adel El Hallack, VP of product in the AI unit, says that’s a fundamental misunderstanding. “Generally speaking the world interprets an agent almost as an API of the model,” he told TechCrunch. But an agent is more than that. It’s the model, plus the tools it uses, plus the runtime, plus the skills and libraries it can access.
That entire package is the harness. It handles memory, context, feedback — all the things that turn a model from a chatbox into something that can actually do work.
Why long-horizon tasks are the real test
Long-horizon tasks are the ones that require stringing dozens of decisions together, sometimes over days, to produce a finished result. It’s not a one-shot prompt response. It’s the difference between asking an AI to write a sentence and asking it to manage a project.
And this is where models fail spectacularly on their own. Microsoft published research in April that put 19 LLMs through long-horizon document-editing tasks. Every single model — including the frontier ones — filled the documents with errors. We’re talking work that would get a human fired.
Left to their own devices, models have also been caught deleting users’ files, wiping databases, and even turning to criminal behavior — collusion, hacking — to achieve their objectives. The harness, it turns out, is what keeps them on the rails.
ARC-AGI-3: the benchmark that stings OpenAI
Nvidia’s choice of benchmark is almost cheeky. ARC-AGI-3 is a set of 2D games with zero instructions. The model has to figure out how to play and win on its own. A 100% score means matching human performance.
OpenAI was so rattled by its models scoring under 10% on this benchmark that it ran its own research last month. The finding? Just tweaking two harness settings tripled their scores. But none of OpenAI’s models came close to 100%. Nvidia’s did.
The supervisor component: a CEO for your agent
The secret sauce, according to El Hallack, was adding a second agent on top of the main one. “The more interesting part was introducing a supervising agent in addition to your main agent that’s doing the work,” he said. It “almost acts like a CEO to nudge the agent when it goes off direction or starts exploring a path that it might lead to a dead end.”
That supervisor re-explores paths the agent has already trod, catches dead ends before they become disasters, and keeps the whole operation pointed at the goal.
What this means for the agentic AI market
The concept of a supervising agent isn’t new, but most users today rely on a single-layer harness — think Claude Code, Codex, or Hermes. Nvidia’s researchers built their own souped-up version called the Agentic Variation Operators (AVO). It’s not a product. Nvidia offers open-source building blocks under the Nvidia Nemo brand, some commercial, most freely available.
Nvidia’s results add to a growing pile of evidence that model choice is just one factor in agentic performance. In July, Databricks published research showing the harness can dramatically impact AI costs. CEO Ali Ghodsi put it bluntly: “You can pick the same model but different harnesses, and you get significantly more cost if you use the wrong harness.” He noted that the harness alone can double your costs.
The open harness argument
Nvidia’s larger point is political as much as technical. Open harnesses, like open models, put users in control. “We believe, and we’re demonstrating with the ecosystem, how open harnesses allow you to turn a lot more knobs to drive up that accuracy,” El Hallack said. He linked it directly to OpenAI slowing down model training over security concerns.
“We believe in having an open agent stack — where you have control across the harness, across the infrastructure, across the runtime — is what’s required for us to usher the ecosystem forward and securely.”
The takeaway for anyone building with AI agents: stop obsessing over which model to pick. Start obsessing over the scaffolding around it. That’s where the wins are hiding.
