换壳不换芯：编码智能体框架的性价比真相
论证要点
- 先前研究宣称框架间差距超20分，但多来自单次运行；同一配置重跑在SWE-bench Verified上已有最多6分波动，智谱宣称的2.4分优势落在噪声范围内。
- 作者用5个模型跑3个框架（Claude Code、OpenCode、mini-SWE-agent），在447个任务上最重与最轻框架差距不到2分，统计上等价；45个最难任务上差距仅2-5题，重跑波动可达3题。
- 重跑与换框架的任务翻转率均为13%，换模型才升至22%；框架差距表现为“同一噪声的重新抽样”，且小样本下多数宣称改进无法可靠检测。
- 唯一可测的框架效应是OpenCode稳定落后最多9分，约一半归因于输出超限时未像Claude Code那样提示继续；其余差距作者无法解释，仅观察到模型更早停止工作。
- 成本差异主要来自系统提示和工具描述：Claude Code约16,600 token、OpenCode约7,000、mini-SWE-agent不足1,000，且每步重发；同一任务成本可差三倍，步数放大基线成本。
- 思考模式是框架选择唯一真正改变结果的场景：关闭思考后Qwen3.6在mini-SWE-agent上掉11分，HY4-Preview在Claude Code上掉11分；作者承认测试网格太小无法解释原因。
对实践的启示
选框架别只看榜单分数，先跑同一配置的重复基线，把噪声范围测出来；系统提示和工具描述的体积直接决定账单，精简提示往往比换框架更省钱；若模型依赖推理，需单独验证关闭思考后的表现。
作者承认的反方与局限
作者承认测试网格太小，无法解释思考依赖为何因模型-框架组合而异；研究主要依赖单一基准，仅用Terminal-Bench 2.1做小规模交叉验证；闭源顶级模型Opus 5只在Claude Code中运行；OpenCode的本地模型时间限制更紧，属未控制变量。
原文摘引
The paper's title gives away the answer: "What does a harness buy? Mostly tokens."
The harness gap behaves like a "fresh draw of the same noise," the authors write.
A harness buys "fewer ways to lose a task, and a bill, not pass rate," the authors conclude.
▸ 展开原文全文（38 段，抓取自 The Decoder）
Fancy harnesses make coding agents more expensive but not better, study finds
Makers tout score gains from new agent harnesses like Claude Code or OpenCode. A study finds that claim doesn't hold up. Swapping harnesses barely moves results more than running the same setup twice, but costs can triple.
A coding agent pairs a language model with a harness, the system prompt, tools, and context management that make the model work inside a code repository. New harnesses ship every day, and leaderboards mix them freely, but two researchers at Shandong University measured how much the harness actually matters when you hold the model constant. The paper's title gives away the answer: "What does a harness buy? Mostly tokens."
Earlier comparisons were measuring noise
Previous studies report gaps of more than 20 points between harnesses, but those numbers almost always come from a single run, according to the authors. Pure reruns of the same setup already vary by up to six points on SWE-bench Verified. Zhipu AI, for example, advertises its own harness with a 2.4-point lead over Claude Code, a margin that falls well within that noise range. Harnesses are also often wired into the benchmark differently, making comparisons even less reliable.
Liu and Han ran five models through three harnesses in their shipped versions: Claude Code, OpenCode, and mini-SWE-agent. mini-SWE-agent gives the model only a Bash tool and skips context management. Claude Code and OpenCode offer specialized file and search tools, long system prompts, and automatic history compression.
The models were Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview. Claude Opus 5 ran as a reference only in Claude Code. Tests used the 45 hardest tasks from SWE-bench Verified plus a pool of 447 more. Several setups ran twice to measure noise.
All runs happened offline. In early tests with web access, 71 percent of Deepseek and Qwen model attempts pulled the correct fix straight from GitHub instead of solving the task themselves. Claude models almost never did this. The researchers discarded those results and don't claim any maker trained on the test data, but the pattern was strong enough that they cut off web access for all remaining runs.
Swapping harnesses moves results about as much as a rerun
On the larger pool of 447 tasks, Claude Code and mini-SWE-agent, the heaviest and lightest harnesses, sit less than two points apart for both tested models. Within a five-point band, they're statistically equivalent.
The researchers expected the biggest harness effect on the 45 hardest tasks. Instead, harnesses differ by two to five solved tasks per model, and a simple rerun of the same setup shifts the count by up to three. Sometimes mini-SWE-agent leads, sometimes Claude Code does.
The flip rates tell the same story. Between two reruns, the median outcome changes on 13 percent of tasks. Between two different harnesses, it's also 13 percent. Swapping models pushes that to 22 percent. Which tasks a given harness wins changes from run to run, matching a coin flip. The harness gap behaves like a "fresh draw of the same noise," the authors write.
These small samples also limit what any comparison can detect. With 45 tasks, even a 13-point gap shows up only half the time. With 447 tasks, gaps of about five points become reliably detectable. Zhipu's claimed 2.4-point advantage falls below both thresholds, and so do many other advertised improvements.
All harnesses perform the same, except OpenCode falls behind
The only measurable harness effect is a steady deficit for OpenCode. On the large task pool, it trails the other two by up to nine points. About half of that gap traces back to an output limit. When a model exceeds this limit mid-step during reasoning, Claude Code prompts it to keep going. OpenCode lets the step end there, and the attempt usually fails.
Adding a Claude Code-style continuation prompt to OpenCode didn't help, because the same task rarely triggers the same overflow on a retry. For the rest of the gap, the authors have no explanation, only an observation: under OpenCode, models stop working much sooner. On locally run models, OpenCode also had a much tighter time limit than Claude Code, which the authors flag as an uncontrolled variable.
mini-SWE-agent runs into the context window because it lacks context management, mostly on tasks where the other harnesses also fail. OpenCode's search tools try to download a binary file on first use and break when offline.
System prompt and tool descriptions set the bill
With the same model, same tasks, and same price list, a single task can cost up to three times more depending on the harness. Tool outputs barely explain the gap. All three harnesses return similar amounts of text per call.
The real difference comes from the system prompt and tool descriptions. They get resent with every step and run about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode, and under 1,000 for mini-SWE-agent. Claude Code also keeps the model's reasoning output in the conversation history, based on the researchers' reverse engineering.
After that first call, context grows at similar rates across all harnesses. The system prompt size sets the baseline cost, and the number of steps multiplies it. Which factor dominates depends on the model: With GLM-5.3-Flash, Claude Code costs nearly twice as much as mini-SWE-agent. With Deepseek-V4-Flash, mini-SWE-agent needs so many extra steps that the two end up even. Lower prices for cached tokens dampen the gaps but don't change the ranking.
No home-field advantage, open questions around thinking
Deepseek's own harness scores no better with its own model than the other harnesses do, but costs the most. Claude Opus 5 delivered identical results even after the researchers removed tools from Claude Code. A year-old version solves a similar number of tasks at less than half the cost, because Extended Thinking wasn't on by default back then.
Thinking was the only case where the harness choice actually changed the outcome. Without thinking, mini-SWE-agent with Qwen3.6 drops eleven points, while Claude Code and OpenCode stay within noise. HY4-Preview also loses eleven points in Claude Code without thinking. How much an agent depends on reasoning seems tied to the specific model-harness combination. The authors admit their test grid is too small to explain why.
The study relies mainly on one benchmark. A small cross-check on Terminal-Bench 2.1 shows the same pattern. The closed-source top model, Opus 5, ran only in Claude Code. A harness buys " fewer ways to lose a task, and a bill, not pass rate," the authors conclude. They recommend publishing a rerun of the same setup alongside every harness comparison. That's the only way to tell how much of a measured difference is just noise. Data and analysis scripts are available on GitHub.
More effort in the harness rarely pays off in other studies, either. In one, a simple Markdown file beat a complex skill system for AI coding agents. An earlier study already showed that the choice of framework can nearly triple costs for the same model. Harness design gets a lot of attention as a result. Nvidia recently cut nearly half its token costs for coding agents through better agent steering.
AI News Without the Hype – Curated by Humans
Subscribe now
Read on for the full picture.
Subscribe for hype-free coverage.
- Full access to every article on THE DECODER
- No ads
- Join the comments and community discussions
- A weekly AI news recap via mail
- 6x/year: "AI Radar" — deep dives on the AI topics that matter most
- Daily AI news, always up to date
- Our full ten-year archive
- Covered by a team with 10+ years in AI