From Leaderboards to Model Profiles: Why Resolve Rate Fails Agentic Coding
发布于 2026-09-03
Resolve rate treats every passing run as equivalent. JetBrains' trajectory-aware evaluation pipeline and per-model behavioral profiles show why that equivalence is fictional — and what should replace it.
From Leaderboards to Model Profiles: Why Resolve Rate Fails Agentic Coding
On August 31, 2026, the JetBrains Junie team published an engineering post that quietly destabilized how the industry compares coding agents. Claude Opus 4.7 and Gemini 3.5 Flash, run inside the same Junie harness across 523 benchmark tasks, solved 267 and 254 tasks respectively — 51.1% versus 48.6% resolve rate. By every public leaderboard in use today, Gemini is the deal. The post then shows that the two models reached that nearly identical score through radically different paths, and that the leaderboard saw nothing of it[^F01].
This is not a model-versus-model take. It is an argument that resolve rate, the metric the entire agentic-coding industry has been sorting by, is the wrong unit of decision.
The numbers hiding behind a tied score
On one of JetBrains' private benchmarks, Opus and Gemini Flash solved exactly the same number of tasks. The tie concealed a 2.25× cost gap and a 1.47× effort gap. Opus averaged 184 steps per run at USD 2.79; Gemini averaged 271 steps at USD 1.24[^F02]. If a team is buying model capacity by the dollar, Gemini wins on this benchmark. If a team is buying a model that will be embedded in an agent that has to work inside a tight step budget or a tool-call rate limit, the answer flips. The leaderboard was silent on both.
A four-model comparison on 522 shared tasks makes the trade-off frontier starker. GPT-5.5 reached the highest resolve rate at 51.5% and was the only model in the set that always ran an executable check. Qwen 3.6 27B FP8 resolved 38.9% — a 12.6-point gap from GPT-5.5 — at roughly 3% of GPT-5.5's per-run cost[^F03]. The leaderboard calls GPT-5.5 the better model. A team running a high-volume refactoring pipeline at a fixed budget could plausibly call it the wrong one.
What "passed" actually means
#AI Agent#AI编程#AI工程#Claude
Resolve Rate Is Failing Agentic Coding: What Comes Next
Resolve rate treats every passing run as equivalent. The JetBrains data shows this equivalence is fictional. In 123 of Opus's 267 resolved runs, the agent performed no executable validation — meaning the patch was declared correct on the basis of the test outcome alone, without the agent ever running the test itself. 68 of those runs still resolved the task on the tests that did execute[^F05]. A patch that never went through validation can still pass when the tests eventually run later, but a patch that was never confirmed-correct in its own run carries undetected risk of regressions and edge-case failures that only running the code would surface. Resolve rate gives no signal about whether the agent validated its own work.
Gemini's profile inverts the problem. Gemini ran executable checks more reliably than Opus, but 195 of its runs — 37.3% — were assessed as containing moderate or severe hallucination, against 130 runs for Opus. 80 of Gemini's runs (15.3%) showed significant or severe redundancy, more than twice the Opus rate of 6.5%[^F06]. A model can validate often and still produce sprawling patches that touch unrelated files. Validation frequency is not patch quality.
The same issue shows up at the wider model comparison. Opus was assessed with moderate or severe hallucination in 24.7% of its runs against 33.7% for GPT-5.5. Patch redundancy was 6.3% for Opus against 13.2% for GPT-5.5[^F04]. The two models finished four tenths of a percentage point apart on resolve rate and within a cent of each other per run, so a leaderboard would mark them interchangeable. Their patches are not.
The pipeline that sees what resolve rate cannot
JetBrains' evaluation pipeline is built to capture what resolve rate discards. For each benchmark task, the pipeline combines the issue, repository context, the generated patch, the test result, and the full execution trace. It then evaluates the run from four perspectives, each asking a distinct question[^F07]:
Outcome: did the patch resolve the task, and which tests passed or failed?
Efficiency: how many tokens, model calls, tool calls, and seconds did the run require, and what did it cost?
Patch quality: did the change touch the relevant files and symbols, stay contained, and avoid unnecessary complexity?
Process quality: how did the agent move through exploration, implementation, and validation? Did it reproduce the problem, repeat work, or stop without testing the final change?
The deterministic layer derives reproducible measurements from logs and repository data — test outcomes, runtime, token use, tool calls, modified files and symbols, code complexity changes, repeated file reads, unchanged command retries, tool failure loops. Rules alone cannot interpret every action: opening a file twice may be wasteful, or necessary after a related edit. A large patch may be unfocused, or appropriate for a change that spans several components. For these questions, LLM judges receive structured evidence from the issue, patch, trajectory, and bounded repository context, and assess milestones: did the agent find the relevant code, reproduce the defect, identify the root cause, address it, introduce unnecessary complexity, and validate the result[^F07].
The combination of the two layers is what makes the pipeline interesting. A purely deterministic layer can count tool calls but cannot judge whether those tool calls were appropriate. A purely LLM-judged layer cannot reproduce its own answers. Stacking them gives a clearer account of progress and a sharper location for failures.
What the profiles say
Run the pipeline across models and the leaderboard stops being the primary signal. Two model profiles from the JetBrains data:
Claude Opus 4.7 is a strong diagnostic model with a known weak spot after localization. It often reached the correct mechanism or architectural layer and solved 53 tasks Gemini missed. The weakness shows up after that: some runs found the right mechanism but stopped with a reproduction test, missed a companion branch or call site, or implemented a plausible custom solution instead of following an existing repository pattern[^F01]. The 123 runs with no executable validation come largely from this profile. Opus is the right starting point when the main challenge is understanding an unfamiliar repository or separating a visible symptom from its source.
Gemini 3.5 Flash is the opposite. It is more likely to run an executable check and use the output to refine the solution. The risks are convergence — repeated searches after reaching the relevant code, equivalent commands repeated, many steps spent on build infrastructure — and repository grounding. Gemini was more likely to rely on unverified APIs, dependencies, paths, or test fixtures. The 37.3% hallucination rate and the 15.3% redundancy rate are the visible form of that grounding gap[^F06]. Gemini benefits from precise task contracts, symbol verification, clear stopping rules, and a final review of the diff.
GPT-5.5 lands in the middle: highest resolve rate, only model that always validated, but also the highest hallucination rate of the four at 33.7% and a patch redundancy rate of 13.2%[^F04]. Qwen 3.6 27B FP8 lands at the cheap corner — 38.9% resolve rate, roughly 3% of GPT-5.5's per-run cost[^F03]. The frontier has a cheap tier; the leaderboard treats it as a worse model.
This is not only a JetBrains problem
The same critique shows up across the industry in the same week. On August 25, GitHub's secret-scanning LLM team published a post on the discipline they used to move from prototype to production. They treated precision as the primary outcome, recall as a safety constraint that an experiment could advance only if any decrease stayed within a predefined acceptable range, and latency, cost, and reliability as operational guardrails[^F09]. Suppressing a real credential is more costly than asking a developer to review one more alert. The post is structurally identical to the JetBrains framing: one metric leads, another metric constrains, the rest guard.
On August 26, AWS released Amazon Bedrock AgentCore Evaluations, designed to score any agent that emits OpenTelemetry — LangGraph, LlamaIndex, OpenAI Agents SDK, Google ADK, Claude Agent SDK, Strands — by reading spans rather than framework-specific SDK calls[^F10]. The shift is from "did the agent pass the test" to "what telemetry did the agent emit, and what does that telemetry say about how it worked." The industry is moving toward trajectory-aware evaluation at the framework level, not just at the benchmark level.
On September 1, Qdrant released Qdrant-FineWeb-10B, a 24.47 TB vector benchmark with exact top-1000 brute-force ground truth for 100,000 queries across a 10-billion-document corpus, explicitly arguing that the era of 1-million-vector synthetic claims is over[^F11]. The critique is the same one JetBrains makes, on different ground: small, opaque benchmarks with synthetic queries are not the ground truth for production systems.
What this means for how teams pick
The right unit of decision is not the resolve rate. It is the model profile — a structured answer to "what does this model do well, what does it do badly, what does it cost, and how does it fail?" Two teams buying agent capacity should reach different conclusions from the same data. A team running greenfield exploration in unfamiliar repos benefits from Opus's diagnostic strength. A team running high-volume refactoring with tight contracts benefits from Gemini's validation discipline. A team running at fixed budget benefits from Qwen's cost position even at the resolve-rate cost.
Profile-driven selection turns the leaderboard into a tiebreaker, not a primary signal. The procurement question becomes: which axis matters for our task class, which model leads that axis, and what does our worst-case failure look like if we pick wrong? The JetBrains data already gives partial answers. The remaining gap is a shared schema for profiles that lets a team compare vendor-published profiles the way it can compare resolve rates today.