One Model, Two Scores 36 Points Apart
On September 3, ARC Prize published a new ARC-AGI-3 result that should have made more headlines. The same model — GPT-6 Astra, at the same reasoning effort — scored 62.7% in a standard harness, then 98.6% in a Provider Adapter harness.
Read that again: this is not a comparison between two models. It is the same brain taking the same exam twice, with a 35.9-point gap. And the counterintuitive kicker: the higher-scoring setup was also cheaper — roughly $26,100 down to $17,300, a 34% cost reduction.
A few days later, YC ran a Paper Club titled "Why the Harness Matters More Than the Model," hosted by YC partner François Chaubard, with Seth Karten (Prime Intellect), Jon Saad-Falcon (Stanford), and YC's own Josh France and Regan Bell. Different researchers, same fixation: everything wrapped around the model.
Most coverage treats this as a benchmark curiosity. It is something more serious: the decisive battleground in AI is shifting from "whose model is stronger" to "who can organize models better." Model capability is becoming a given. Under identical conditions, agents still differ by 36 points. Those 36 points are the value space of the harness — and the most underpriced moat in the industry.
The Model Sets the Ceiling; the Harness Decides Whether You Reach It
A harness, literally, is what you strap onto a horse to transmit its power to the cart. In AI terms, it is the entire operating framework around a model: how the system prompt is written, how context is organized, which tools are available, where memory lives, how sub-agents divide work. The model decides how smart the brain is; the harness decides what that brain has in its hands while working.
Chaubard lived this shift. In March, while trying out Karpathy's automated research project, he initially just wanted an interface to watch what the agent was doing. Then retrieval, experiments, review, writing, and progress management kept getting wired in — and he realized he had accidentally built a full harness. Now he hands the system a research direction and an evaluation metric, and multiple agents grind away. The models never changed. The deliverables kept getting better.
Worth noting the stigma: prompt tweaking, tool plumbing, and task loops were long considered second-class work in machine learning — engineering, not real research. That "menial" layer is now directly moving scores.
It is not that models got weaker. It is that once models got strong, the remaining gaps all live outside the model.
Three Rungs: From Prompt Tuning to Rewriting the Harness Itself
Harness optimization is climbing a clear ladder. Three rungs matter.
First, automated prompt optimization. Systems like DSPy try different prompt formulations and pick winners using test results — turning prompt tuning from a craft into a search problem.
Second, rewriting the harness code itself. The Darwin Gödel Machine (DGM) does not just tune prompts; it edits the agent's own framework code and keeps whichever version scores better. In the paper's experiments, it lifted SWE-bench performance from 20% to 50% — more than doubled, with the model unchanged.
Third, experience accumulation. Continual Harness lets agents review past task records and decide whether to revise prompts, add skills, update memory, or reconfigure sub-agents. Lessons from previous runs start getting baked into the next version.
The frontier is now pushing further: retraining on the agent's own freshly generated data, even updating weights at test time. The optimization frontier is migrating from the harness into the model — the boundary between harness and model is dissolving.
Give this dynamic a name: orchestration-layer arbitrage. As models converge on capability, whoever operationalizes the layer above them extracts more delivered value from the same-priced model. Structurally, it resembles the cloud-era infrastructure playbook: the model is compute, the harness is the orchestration layer — and profits historically settle in orchestration.
Three Proofs: Long-Horizon Agents, Local AI, and the Enterprise
The framework earns its keep across three very different settings.
Long-horizon tasks. Prime Intellect's Prime Agent refuses to script every step. Instead it layers information: what matters now goes in context; programs, results, and progress live in a persistent REPL; long-term memory and skills sit in external storage. The team ran an agent on Factorio for seven straight days — 633 sub-agent calls, over 23 million output tokens — completing 24 of 196 technologies and pushing the next one to 71%. At one point progress collapsed from five completed technologies back to one. The system did not wipe and restart; the agent re-planned from preserved state. For long-horizon work, the core asset is not the plan. It is continuity.
Local personal AI. Stanford's OpenJarvis wants to move personal AI onto your own device, but local models trail frontier models by roughly 6–12 months of capability. The team's blunt experiment: swap Claude Opus 4.6 for Qwen3.5-9B in OpenClaw and Hermes Agent setups, change nothing else — accuracy dropped 24.8 and 38.8 points on two benchmarks. Then they optimized the whole system across five layers, letting the cloud model analyze failure cases and tune the local configuration while the local model does the daily running. Result: Qwen3.5-9B unchanged, yet the system recovered about 56% and 77% of those losses. Across eight benchmarks, the best local setup averaged 80.3% versus Opus 4.6's 83.5% — a 3.2-point gap — at roughly 1/800th of the cloud's marginal API cost.
The enterprise. YC once ran 50+ Hermes Agents across virtual machines, each configured individually, each debugging session a login-into-the-box affair. Their internal harness, QM, centralizes conversations and state while virtual machines become on-demand resources. Scale then surfaced three harder problems. Agents mistake local problems for global ones — the team calls it "main character syndrome," and still gates such changes with human review. Agents give up too early — YC built a "Grind Tool" enforcing minimum runtime or token budgets. And the hardest: permissions. An agent reading Slack has access to context it does not understand well enough to relay. YC's current answer: databases default to read-only, writes require human sign-off — though as trust grows, human approval risks degrading into rubber-stamping.
All three converge on one caveat: the harness amplifies capability that already exists; it does not conjure it. The same Prime harness scored 95.5% with Claude Opus 5 on the ARC-AGI-3 public set — and 25.7% with Terra. Nearly 70 points of spread, same harness, different brain. The model is still the foundation. The gap, increasingly, is built on top of it.
What To Do: Move Budget From Model Swaps to Harness Work
For two years the industry bet nearly everything on models. The cheapest leverage now sits elsewhere. Depending on your seat:
- If you are a technical decision-maker: run a harness baseline audit before your next model swap. Test the same model under two harnesses, as ARC-AGI-3 did. If your spread is under 5 points, your orchestration layer has never been seriously optimized — that is usually the largest free lunch available.
- If you build agents: make information layering the default architecture — context holds only what is current, state lives in a persistent environment, memory sits in external storage. And enforce minimum task budgets (the Grind Tool pattern) so your agent does not surrender on attempt three.
- If you deploy agents in an enterprise: fix permissions before capability. Read-only database defaults and human-approved writes are the floor — but design against approval fatigue now, with sampled human re-review on high-risk operations rather than blanket click-through.
- If you are a founder: do not fight at the model layer. The orchestration-arbitrage window is now. The day models fully converge, the harness difference becomes the entire product difference.
The model decides how fast you can go. The harness decides whether you can keep going at all — and how far.
Sources: Facts and figures in this article are drawn from "Silicon Observer Pro"'s coverage of YC's Paper Club "Why the Harness Matters More Than the Model" (republished by 36Kr), including ARC Prize's September 3, 2026 ARC-AGI-3 results, Prime Intellect's Prime Agent, Stanford's OpenJarvis, and YC's internal QM harness.