final

AOE Tech Labs published a benchmark this month. Same DeepSeek-V4-Flash model. Different Harness. Up to 23.6% gain on long-horizon agent tasks. And a 57.1× price gap closed against Claude Opus 4.8.

This is the first clean empirical confirmation of what the field has been whispering for a year: for real Agent work, the model weights matter less than the system around them.

The 57.1× price gap

DeepSeek-V4-Flash 0731 lists at $0.14 input / $0.28 output per million tokens. At a typical 3:1 agent ratio, that's $0.175 per million blended.

Claude Opus 4.8 lists at $5 / $25. Same 3:1 ratio: $10 per million blended.

That's 57.1× in linear terms, not log scale. AOE Tech Labs did not pick the cheaper model to be cute — they picked it to prove a point. With the model locked, only the Harness varies. Whatever difference shows up belongs to the system, not the weights.

The five-benchmark scoreboard

Five third-party agent benchmarks, all run with the same DeepSeek-V4-Flash 0731 base model:

Benchmark	DeepSeek's own Harness	Floatboat Harness	Opus 4.8 baseline
DeepSWE	54.4	67.25	58.0
Terminal Bench 2.1	82.7	82.7	82.7
SWE-bench	73.2	80.1	below 80.1
Toolathlon	25.1	28.2	below 28.2
BrowseComp	70.7	84.7	below 84.7

Floatboat's report beats Opus 4.8 on all five. The single-variable design — same model weights, same API, isolated sandboxes — makes the gap hard to wave away.

The interesting row is Terminal Bench 2.1: tie at the top (82.7 = 82.7). The two systems converge on tasks that have a clean definition and a clean answer, then diverge as soon as the task has more steps than the prompt can hold.

What a Harness actually is

Three pieces, none of which the model controls:

1. What the model can touch. Files, APIs, browsers, terminals, calendars. Without an executor, GPT-5 is a typewriter with no paper.
2. How each loop closes. Tool-call parsing, error recovery, retry policy, schema validation. The boring 90% of agent reliability.
3. Where state lives. Conversation history, scratchpads, memory across turns, plan-and-replan. Anything longer than one context window dies here.

A model that can't see your filesystem can't edit your file. A model that can't remember what it tried last turn can't recover. None of those are model problems — they are Harness problems.

The 1.9% → 23.6% curve

The cleanest data point in the report is the gain curve. Same base model, same vendor, same prompts. Only the Harness changes:

• 1.9% on the shortest task
• 9.6% on a mid-length chain
• 12.6% on a longer chain
• 19.9% on a multi-tool chain
• 23.6% on the longest, most stateful task

The longer the chain, the more the Harness earns. This is the rule worth memorising: Harness gains scale with task length.

Real work is long-horizon. Real work is "open this spreadsheet, find the row, fix the formatting, save the new version, then email the report." That is not a one-shot task. That is a 40-step plan with state, retries, and recovery. The Harness is what carries it.

Why the curve goes up — the mechanics

A short task is a single model call. The Harness mostly does nothing: one prompt in, one answer out, no state to manage. A 1.9% gain there is real but cosmetic — the model dominates.

A long task is a different animal. Five places where the Harness earns its keep:

• Tool-call parsing. A model emits malformed JSON, the schema validator rejects it, the retry policy triggers. Cheap models produce malformed tool calls more often than expensive ones — the Harness's job is to catch every one of them and re-prompt cleanly.
• State reconstruction. After 30 turns the original system prompt is a memory. The Harness rebuilds the working context each turn from a structured plan store. Without that, the model loses the thread.
• Error recovery. A web fetch returns 503. The Harness decides: retry with backoff, switch to a fallback, or escalate to the user. The model cannot decide this — it has no clock, no budget, no concept of failure.
• Multi-modal tool switching. Read a PDF, run a SQL query, render a chart, attach to email. The Harness routes between tool families and translates results back into text the model can reason about.
• Verification. A 40-step plan needs a final check. The Harness spins up a "reviewer" pass — sometimes with a different model — and either accepts the result or kicks it back into the loop.

Stack all five and a $0.175 base model on a great Harness looks like a $10 model on no Harness. The work the Harness does is invisible until it isn't.

The industry has already moved

Three pieces of cross-evidence that this is not a one-vendor story:

• Microsoft released Agent Framework Harness and Hosted Agents on 2026-08-12. The platform vendor has decided the Harness is a product surface, not an implementation detail.
• DeepSeek launched its own "DeepSeek Harness" team on 2026-08-13. The model vendor has decided to compete on Harness quality. Even the cheapest model maker is racing here.
• LangChain's Harrison Chase and YC's Garry Tan publicly debated "Why models can never 'eat' the Harness" in August. The 51.2K lines of code in LangGraph are the proof — the Harness is where the work lives, and it does not compress into a 200B parameter model.

The bias question, named

Yes, Floatboat published its own benchmark. Yes, vendor-published benchmarks have a history of being flattering. Two things reduce the surface area here:

1. Single-variable design. Same weights, same API, same sandboxes. The control is the model vendor's own Harness — DeepSeek's engineers, not Floatboat's — so the comparison isn't "product vs naked API."
2. Conservative checkpoint. AOE Tech Labs used the 0731 release, not a Preview checkpoint. The Preview run would have shown a +821% gain on DeepSWE that looks too good — and they deliberately didn't put it on the headline number.

This is still vendor data. Treat the magnitudes as directional, not absolute. But the direction is the same direction the rest of the industry is moving in.

What this means for the next 12 months

Three predictions, all downstream of one observation:

1. Pricing pressure collapses toward the cheap model. If a $0.175/M base plus a serious Harness beats a $10/M base on long-horizon work, the premium for frontier model weights gets thinner by the quarter. The market will adjust.
2. Harness engineering becomes a job title. "Harness engineer" sits between ML engineer and platform engineer. The people who can ship a working multi-turn loop on a tight model are about to be very employable.
3. The benchmarks shift. Five years of benchmarks focused on model quality (MMLU, GSM8K, HumanEval). The next five will be Harness benchmarks — task-completion under real agent constraints, not single-turn QA.

The thing nobody is saying out loud

The reason this matters more than any single benchmark is what it implies about the last two years of model releases.

Most frontier-lab marketing has been a model-weight story: GPT-5, Opus 4.8, Gemini 3, Claude Mythos. Each one claimed to eat more of the Harness. Each one delivered a smaller and smaller fraction of agent reliability gain.

The reason is the rule above. Model improvements help short tasks and one-shot tasks. Long-horizon tasks need Harness improvements. The two are not interchangeable.

If you're shipping an Agent product, your bottleneck is almost certainly not the next model release. It's the system you're wrapping it in.

What this looks like when you build one

Three traps teams fall into when they treat Harness as an afterthought:

• Treating the prompt as the product. The model and the prompt are one layer. The Harness is the other four layers. Teams that ship a great prompt and a brittle Harness spend weekends watching their agents fail at step 3.
• Optimising for the wrong benchmark. MMLU scores do not predict SWE-bench Verified scores. Agent reliability scores do — and most teams aren't measuring them yet. Start measuring on a long-horizon benchmark that looks like your real workload, not a single-turn QA set.
• Letting the vendor do it for you. Claude Code, Codex CLI, and the various "agent platforms" ship with their own Harness. If your workload doesn't match their assumptions, you inherit their ceiling. The teams shipping differentiated agent products in 2026 own their Harness end-to-end.

The math is unforgiving. A vendor-managed Harness on a frontier model costs $10/M and gets you a fixed feature surface. A custom Harness on a cheap model costs $0.175/M plus engineering — and gets you whatever feature surface you build.

---

Sources: AOE Tech Labs, Floatboat Harness Benchmark Report (https://floatboat.ai/news/harness-benchmark), 2026-08-07. Vendor pricing: DeepSeek API, Anthropic Claude pricing pages. Industry corroboration: Microsoft Agent Framework Harness launch (2026-08-12), DeepSeek Harness team launch (2026-08-13), LangChain / YC public debate (2026-08).