Your Agent Writes 47% of Your Code. Who Checks the Tests It Wrote for Itself?
Published on 2026-09-01
When 47% of code is agent-written, the tests standing next to it are usually agent-written too. Most of those tests pass without testing anything. Mutation-checked tests change that — and a 66.7% clean-decline rate shows why 'verified' isn't the same as 'looked at it once.'
Your Agent Writes 47% of Your Code. Who Checks the Tests It Wrote for Itself?
If your repo still ships with handwritten tests, you are not paying attention to 2026. JetBrains' latest developer survey puts 47% of code written by an AI coding agent. The follow-up question nobody on that survey asked is the obvious one: what percentage of those tests actually test the thing they're supposed to test?
Most of them don't. They look like tests. They run. They go green. And the green checkmark says nothing different than a red one would have, because neither was ever possible.
That gap — between "looks done" and "is done" — is what a small open-source tool called spec-verify exists to close. Here's what it caught on a real dedup function, on its author's own code, on its first run.
What a vacuous test actually is
A vacuous test passes no matter what the code does. The classic shape:
deftest_dedup_sessions_runs(tmp_path):
result = dedup.dedup_sessions([str(f)])
assert result isnotNone
It calls the function, gets a list back, passes. It would also pass against a version of dedup_sessions that used the filename as the deduplication key — the exact bug the assumption flagged — because the test never checks which sessions survived dedup, only that something came back.
That second version isn't hypothetical. It's the bug the test was supposedly guarding against. And the test doesn't notice.
A vacuous test looks exactly like every other passing test in the suite. That's the trap. Green checkmarks stop being evidence the moment you can't tell a vacuous test from a real one.
Worse: these tests don't just fail to catch the original bug. They keep passing after the behavior they were supposed to guard breaks for a second, unrelated reason. Nothing gets verified. The test just looks like coverage.
Mutation check — how spec-verify flips it
For each Given/When/Then block in a spec, spec-verify does four things:
Generates the test. The Then clause becomes a concrete assertion on a return value, on state, or a side effect. Never "runs without throwing."
Generates one targeted mutation. Not a general mutation-testing sweep, but one deliberate break informed by the criterion's paired [ASSUMPTION: ...] tag. If the assumption named the risk, the mutation reintroduces exactly that risk.
Runs the test against the mutant. If it still passes, the test never checked the thing it claimed to. That's a vacuous test, and it gets flagged, not trusted.
Fails closed. A criterion whose test can't be validated this way blocks "done" — loudly, by name, with the reason stated — instead of quietly passing review.
Mutation testing isn't new. PIT for Java, MutPy for Python, Stryker for JS have been around for years. The reason PIT never caught on outside Java shops is overhead: full mutation sweeps take 4×–10× the time of the regular test run, and the signal-to-noise ratio is bad. Most mutants don't reflect realistic bugs.
The shift here is pairing one test with one targeted mutant, derived from the assumption that already named the risk. One criterion, one test, one mutation. Three steps instead of a thousand. The signal is the same — does the test fail when the bug comes back? — but the cost is something you'd actually run on every PR.
Here's what it caught on a real dedup function. Two tests, same criterion, one passes as verified, the other as vacuous:
test
baseline
mutant
verdict
test_dedup_verified.py
pass
FAIL
VERIFIED
test_dedup_vacuous.py
pass
pass
VACUOUS
Same green checkmark before you looked closer. A completely different amount of protection.
VACUOUS vs UNVALIDATABLE — two opposite kinds of "can't test"
Not every criterion is mutation-testable. When the fix this article originally shipped — an entity-grounding rule telling the model not to fabricate a payment provider's docs under another provider's name — lived entirely in the system prompt, there was no function boundary to call and assert on. The only way to check it was to ask the model trick questions and read what it says. A mutation test can't reach that.
spec-verify calls this UNVALIDATABLE. It's tempting to treat it the same as a vacuous test: something not-quite-good-enough that needs fixing. It isn't.
A vacuous test is a defect: the test is bad, and the fix is always the same — write a better one. UNVALIDATABLE means the criterion sits outside what this technique can check at all, usually because the behavior is non-deterministic rather than because anyone did anything wrong.
Treat them the same and you ship one of two flavors of broken: rubber-stamp tests you never caught because "some things just can't be tested" (they can, this one just wasn't written to), or work that blocks forever on anything non-deterministic, which on a real codebase is often.
So the gate has two tracks:
VACUOUS and BROKEN-TEST: no waiver, ever. The only way past a defective test is a better test.
UNVALIDATABLE: clears only through an explicit, on-record human sign-off, where a person states in writing how they actually checked it.
The 66.7% problem — when "verified" was honest and still wrong
This is the softer version of the vacuous-test problem, and it's the one that should keep tech leads up at night.
The spec-verify author signed off an entity-grounding criterion with a note citing a real number: zero out of five fabricated responses. That note passed spec-verify's structural check — it wasn't empty or a one-word rubber stamp, and it named an actual method.
It was also, it turned out, an optimistic single run. Independent retesting at a fixed temperature with no fixed seed found a 66.7% clean-decline rate, not 100%. Some provider pairs retrieved near-identical chunks and fooled the self-check more often than the first run suggested.
The honest fix wasn't a better sign-off. It was replacing the thing being signed off: a deterministic code-level gate for the known problem cases, which is testable the normal way, sitting alongside the prompt-based check for everything else.
UNVALIDATABLE isn't a permanent state. It's a flag that something needs a human, or, ideally, needs to stop needing a human at all.
For a team setting, the structural check on sign-off notes (reject empty notes, reject anything under a sentence's worth of text, reject a blocklist of stock phrases like "looks fine" or "lgtm") is the floor. The stronger version: resolve who-signed-off-and-when from a git commit's actual author and timestamp instead of trusting free-text fields in a JSON file. A fabricated sign-off then requires an actual commit under someone's real identity, visible in history, not an edit in a file nobody's watching. Overkill for a solo project. The right call the moment more than one person could plausibly have a reason to fake one.
The "looks done" gap
If 47% of your code is agent-written, the test that agent wrote for itself is the only thing standing between you and a regression you didn't author. The default path — generate code, generate a test, run the test, ship — assumes the test means something. It usually doesn't.
spec-verify is a small piece of a larger pattern. The author's trilogy:
spec-writer catches the feature you built for the wrong reason (writes Given/When/Then + flags [ASSUMPTION: ...] for every decision the agent made without being told to).
spec-verify catches the test that would have told you, and didn't.
git-attributed sign-off catches the rubber-stamped review that passes anyway.
Between the three: an assumption gets flagged, then corrected before the code ships, and now there's a test that actually fails if the correction gets undone six months from now by someone who never read the original spec.
None of this replaces judgment. A spec can be well-formed and still wrong about what to build. A sign-off can be honest and still miss what a harder retest would have found. What all three layers do is make sure the gap between "looks done" and "is done" has to be crossed on purpose, with the reason written down — not skipped because nothing was checking.
What to do this week
P0 — Try the example yourself. Clone spec-verify, run python run_proof.py in the example/ directory. You'll reproduce the dedup table exactly. Five minutes, no API key.
P1 — Pick one vacuous test you shipped last month. Read it. Ask: if the original bug came back, would this test notice? If the answer is "no," that's what spec-verify is for.
P2 — Add a vacuous-test gate to your CI, even a manual one. The minimum viable version: for each PR, run each new test against the diff that introduced it. If the test still passes when the implementation is broken, block the merge.
The honest question isn't whether to trust AI-written code. It's whether the test standing next to that code is doing anything other than looking like a test.
When the agent writes 47% of code AND 47% of tests AND writes the verification — who's in the loop, and at what step? If you can't point to a step, the gap is wider than your green checkmark admits.
Sources: How to Stop Letting AI Agents Fake Their Own Tests (dannwaneri, FreeCodeCamp, 2026-08-26). JetBrains 47% figure per second-hand summary of 2026 developer ecosystem survey; original report link not independently verified.