The Pelican Benchmark Finally Broke. That Is the Result.

The Pelican Benchmark Finally Broke. That Is the Result.

On April 16, 2026, a 20.9GB quantized model running on a laptop drew a better bicycle than Claude Opus 4.7 drew on Anthropic's servers.

Simon Willison ran his pelican-riding-a-bicycle SVG test against two models released the same morning: Qwen3.6-35B-A3B from Alibaba, quantized by Unsloth to a Q4_K_S GGUF and served through LM Studio on a MacBook Pro M5, and Claude Opus 4.7 from Anthropic. He gave it to Qwen. Opus "managed to mess up the bicycle frame" (simonwillison.net, Apr 16 2026). He re-ran Opus with thinking_level: max. The frame came out wrong in a different way.

Then he did the part that matters. Suspecting the labs might be training on a benchmark he has been publishing for eighteen months, he burned a held-out test he had never posted: a flamingo riding a unicycle. Qwen won that too, with sunglasses, a bowtie, and an <!-- Sunglasses on flamingo! --> comment in the SVG source. His conclusion: "I don't think Qwen are cheating."

So the local model won clean. And the person who ran the test immediately told everyone not to believe it: "I very much doubt that a 21GB quantized version of their latest model is more powerful or useful than Anthropic's latest proprietary release."

Both statements are true. That combination is the interesting part.

The eighteen months when the joke worked

Willison started drawing pelicans in October 2024 as a bit. The task is absurd on purpose — a running commentary on how silly model comparison had become. The first batch was, in his word, junk.

Then the joke kept working. Pelican quality climbed alongside everything else. By February 2026, Gemini 3.1 Pro was producing SVGs he described as "illustrations you could actually use somewhere." For a year and a half, an intentionally stupid test tracked real capability well enough that people started quoting it seriously, which annoyed its author.

The reason it worked is worth stating plainly. Drawing a recognizable bicycle in raw SVG, with no canvas and no feedback, demands a lot at once: knowing what a bicycle looks like, holding that shape in mind while emitting coordinates, and keeping the whole thing coherent across a few hundred tokens of markup. Through 2025, models that could do that were also models that could do most other things. One cheap probe, broad coverage.

A proxy works right up until the thing it points at moves somewhere else.

Where the capability actually went

Read what Opus 4.7 shipped with, and the divergence stops being mysterious.

Anthropic optimized it for long-running asynchronous agents and multi-step tasks. It gains most on knowledge-worker work "particularly where it needs to visually verify its own outputs." It calls image-processing libraries programmatically to analyze charts and figures, down to pixel-level transcription. Agents that keep a structured memory store across turns get "more reliable recall and fewer dropped facts without additional prompting." There is a new xhigh effort level, and a taskBudget parameter that shows the model a countdown of its remaining tokens so it can prioritize early and wind down gracefully as the budget drains (Vercel AI Gateway changelog, Apr 15 2026).

Now count how many of those a single SVG request can observe.

Long-running asynchronous execution needs more than one turn. Cross-turn memory needs more than one turn, by definition. taskBudget only does something once there is a budget large enough to run out of. Visual self-verification — the headline gain — requires the model to look at its own output and revise, which is exactly what a one-shot prompt forbids. You ask for an SVG, it emits an SVG, you close the connection. The capability Anthropic spent the release on is structurally unobservable in the test.

The Qwen release points the same direction. Qwen 3.6 Plus lists stronger agentic coding from frontend work to repository-level problem solving, a 1M context window, and better long-horizon planning (Vercel AI Gateway changelog). Both labs are optimizing for the same thing. Neither is optimizing for the first thing they say.

Frontier labs stopped shipping better one-shot outputs. They started shipping better loops.

What a one-shot test cannot see

A single-turn eval measures the quality of one artifact produced from cold context. That is a real property, and for some jobs it is the only property that matters. If you need an SVG of a pelican today, Willison's finding is directly actionable: run Qwen on your laptop.

For anything with a loop in it, the same number tells you close to nothing. Multi-turn work is dominated by failure modes that a first draft never exposes. Does it notice its own mistake on turn four? Does it still remember the constraint you gave it on turn two? Does it recover when a tool call returns an error, or keep calling the same broken tool? Does it degrade sensibly when it is running out of room, or produce three-quarters of a task and stop?

A model can be mediocre at the first draft and excellent at the fourth revision. Another can nail the first draft and fall apart the moment it has to hold state. Both ship. Both post benchmark numbers. Only one of them will survive contact with your agent harness, and the leaderboard will not tell you which.

This is why the vibes and the scores have started to disagree so loudly. People running long agent sessions report clear differences between models that sit within a point of each other on public evals. Neither group is lying. They are measuring different regimes, and the industry has quietly moved from one to the other without changing its instruments.

The uncomfortable version

There is a reading of the pelican result that is more flattering to local models than the evidence supports, and it is spreading faster than the caveat.

A 20.9GB quantized model beating a frontier release on a visible, checkable task is genuinely remarkable, and open-weight models have closed an enormous amount of ground. But the specific gap Willison identified runs the other way. His result is evidence that the benchmark decoupled, not evidence that Qwen surpassed Opus. He said so in the post. The screenshot travels; the sentence under it does not.

The honest summary is narrower and more useful. On single-shot SVG generation as of April 2026, a laptop-sized open model is competitive with, and sometimes better than, a frontier proprietary model. On sustained agentic execution, nobody in that thread produced a number either way, and I am not going to invent one. The published head-to-head does not exist yet.

When a benchmark stops correlating with usefulness, the benchmark did not get harder. Your question changed.

What to do instead

Stop shopping for a general-purpose "best model." That category is dissolving, and the pelican is the visible symptom.

Build a small eval on your own loop. Take three tasks your system actually runs, with the tool calls, the retries, and the context you really pass. Run each candidate model through them ten times. Log where it fails, not just whether it passes — a model that fails at turn six for a reason you can patch is worth more than one that fails at turn two silently.

Measure across turns, since that is where the money goes. Token spend, wall-clock time, and how many turns it takes to reach a correct answer will separate models that look identical on a leaderboard.

Keep the cheap probes as probes. Willison's benchmark is still doing useful work — it just told us something about measurement rather than about models, which is a more valuable result than another point on a chart.

The labs have made their move. Opus 4.7 ships a token countdown so the model can plan its own retreat; Qwen 3.6 ships a million tokens of context and repository-level coding. Both are betting that the unit of work is a session, not a response. The evaluation stack is still built almost entirely around the response.

That gap is where the next eighteen months of confusing benchmark results will come from. The pelican broke first because it was the most honest test we had — a joke nobody could optimize around, which is why it kept working, and why its failure is worth more than most passing scores.