For years the agent community has repeated one equation: Model + Harness = Agent. The left half gets measured constantly — every model launch ships with a benchmark table. The right half — everything beyond the weights: what tools the model can reach, how each loop converges, what the state looks like, where it lives — has never had a ruler.
On August 7, Floatboat (AOE Tech Labs) put that missing half on the scoreboard. Its Floatboat Harness was benchmarked on five third-party evaluations using the cheapest mainstream model money can buy, DeepSeek V4 Flash. It beat a fleet of top-tier overseas models on all five. Against Claude Opus 4.8 — 57.1x more expensive per token — it won every single test.
That is not a “cheap model is actually good” story. It’s the first clean measurement of how much the harness actually contributes. And the setup was deliberately built as a single-variable experiment.
The numbers
Floatboat ran DeepSeek-V4-Flash (official API: $0.14/M input, $0.28/M output), blended at a 3:1 input-output ratio for agent workloads → $0.175/M. The opponent: Claude Opus 4.8 at $5/$25, $10/M blended. On five agent benchmarks — including software engineering, terminal-based tasks and tool use — the $0.175 system outscored the $10 system across the board.
Here’s the part that rules out the obvious reading. The same DeepSeek-V4-Flash, running inside DeepSeek’s own harness, scored 54.4 on DeepSWE — below Opus 4.8’s 58.0. On Floatboat, the same model hit 67.25. On Terminal Bench 2.1, the official harness only tied Opus 4.8 (82.7 : 82.7) and lost the other four. Same weights, same unit cost. The only variable: the harness.
Methodology matters here because the whole conclusion rests on isolating that variable. Both sides used the same DeepSeek-V4-Flash 0731 base; DeepSeek’s public Code Agent benchmarks ran under the official DeepSeek Harness (minimal mode, max settings, top_p=0.95, temperature=1.0); Floatboat ran under its evaluation harness with model inference largely served by DeepSeek’s own API; all tasks ran in isolated, out-of-the-box environments.
The compiler moment
The right analogy isn’t “cheap model finally caught up.” It’s the compiler. Same silicon, same instruction set — and for decades, the thing that turned raw hardware into real performance was the compiler toolchain. Two systems on identical CPUs could differ by multiples on real workloads because one had a mature toolchain and the other didn’t. Agent performance is shaping up the same way: the model is the silicon, the harness is the compiler. Nobody would claim the compiler “is” the chip; nobody should claim the harness “is” the model. But both are where a disproportionate share of real-world performance actually gets decided.
What makes this benchmark worth reading isn’t that one company won. It’s that the field finally has a ruler for the right half of the equation. The industry agreed on the equation years ago; it just never reported the score for one side. Harness design choices — what the model can touch, how tool returns are structured, how loops converge, where state persists — are now measurable, comparable, improvable.
What this shifts
Three practical consequences.
First, the model price war just got recontextualized. If a $0.175/M model plus good scaffolding delivers flagship-level agent results, then “which model” becomes a second-order decision. The first-order decision is which harness you run it in. Teams burning budget on premium weights may be solving the wrong variable.
Second, agent infrastructure is now an explicit competitive dimension. Expect harness benchmarks, harness-as-a-product, and harness moats to become as contested as model leaderboards — the same way Floatboat’s earlier releases (agent-native desktop workspace, agent office networking, schedule-driven agents) turned out to be harness products wearing product-shaped clothes.
Third, measurement itself is leverage. Whoever owns the yardstick for harness quality gets to define how the field optimizes. The report is open: floatboat.ai/news/harness-benchmark, full methodology included.
What you can do Monday morning
- Build your own single-variable harness eval: fix the model, vary the scaffold, measure. One week of this beats a month of model shopping.
- Put agent evaluation in CI, not just RAG accuracy numbers. Track task-completion, loop-convergence and cost-per-completed-task.
- Before upgrading to a pricier flagship, run your current model through a stronger harness first. The cheaper lever is usually the better one.
- If you’re building an agent product, treat the harness as the product. It’s the half of the equation users actually feel.
The takeaway is uncomfortable and useful in equal measure: the gap between models is getting cheaper to close, and the gap between harnesses is where the real separation is happening.