The Pelican Benchmark Finally Broke. That Is the Result.

On April 16, 2026, a 20.9GB quantized model running on a laptop drew a better bicycle than Claude Opus 4.7 drew on Anthropic's servers.

Simon Willison ran his pelican-riding-a-bicycle SVG test against two models released the same morning: Qwen3.6-35B-A3B from Alibaba, quantized by Unsloth to a Q4_K_S GGUF and served through LM Studio on a MacBook Pro M5, and Claude Opus 4.7 from Anthropic. He gave it to Qwen. Opus "managed to mess up the bicycle frame" (simonwillison.net, Apr 16 2026). He re-ran Opus with thinking_level: max. The frame came out wrong in a different way.

Then he did the part that matters. Suspecting the labs might be training on a benchmark he has been publishing for eighteen months, he burned a held-out test he had never posted: a flamingo riding a unicycle. Qwen won that too, with sunglasses, a bowtie, and an <!-- Sunglasses on flamingo! --> comment in the SVG source. His conclusion: "I don't think Qwen are cheating."

So the local model won clean. And the person who ran the test immediately told everyone not to believe it: "I very much doubt that a 21GB quantized version of their latest model is more powerful or useful than Anthropic's latest proprietary release."

Both statements are true. That combination is the interesting part.

The eighteen months when the joke worked

Willison started drawing pelicans in October 2024 as a bit. The task is absurd on purpose — a running commentary on how silly model comparison had become. The first batch was, in his word, junk.

Then the joke kept working. Pelican quality climbed alongside everything else. By February 2026, Gemini 3.1 Pro was producing SVGs he described as "illustrations you could actually use somewhere." For a year and a half, an intentionally stupid test tracked real capability well enough that people started quoting it seriously, which annoyed its author.

The reason it worked is worth stating plainly. Drawing a recognizable bicycle in raw SVG, with no canvas and no feedback, demands a lot at once: knowing what a bicycle looks like, holding that shape in mind while emitting coordinates, and keeping the whole thing coherent across a few hundred tokens of markup. Through 2025, models that could do that were also models that could do most other things. One cheap probe, broad coverage.

A proxy works right up until the thing it points at moves somewhere else.