Would your AI-written tests catch the next bug?

Say an AI wrote thorough tests for a shop's checkout math, and they all pass. Later a teammate's refactor, or an AI agent's edit, makes a bulk discount round a cent high, and those tests are meant to catch it. In one real run on the sample, a scenario file, the AI's first try had 38 tests, and the 35 that passed on the real code caught 8 of 10 small deliberate changes like that one.

Both misses were roundings: a bulk-discounted line price rounded up instead of to the nearest cent, and a percent coupon rounded down. All 35 tests were green, and green says little. Coverage only records which lines ran, so a test with no assertions still counts, and for AI-written suites a 2026 study found coverage near its ceiling.

To find out whether tests would notice, a tool makes one small deliberate change to a copy of your code, the kind of slip a later edit could make, such as a >= turned into >, and reruns your tests. Each such change is a mutant, and one that survives means you're missing a test. That's mutation testing, first proposed in 1971. Anthropic's Claude Code guide says the agent doing the work shouldn't be the one grading it, so when an AI writes your tests, let a script's count of mutants decide when they're done, not the model.

How teams use it#

Stryker is a free, open-source mutation testing tool for JavaScript and TypeScript, C# and Scala, and PIT is its Java counterpart. Stryker works in a sandbox copy of your project, so your real files stay as they are. Each surviving mutant in its report points at a line whose tests would miss a change, so you add or tighten a test there. PIT advises running it often on just the changed code, including on pull requests, and Stryker can too, for faster CI runs. At Google, 6,000 engineers used it on the code changes they wrote or reviewed, as of 2018.

Mutants stand in for real bugs well. A 2014 study of 357 real faults in 5 open-source Java programs found that catching mutants is significantly correlated with catching real ones, independently of coverage.

Reading the tests is no substitute. Two comments in Roscoe's own code record tests that stayed green with the code they covered deleted: in one, every assertion off Windows compared false with false. Nor is asking for thoroughness, since the run's first prompt already used the word.

Six steps and a loop#

The demo calls its mutants planted bugs.

1. 01

   Write the testsAI

   Haiku gets the file and a plain ask for thorough tests, with no word about mutants.

2. 02

   How many planted bugs slip past?Script

   Runs the tests on the real file, where a failing test doesn't count and goes back to be fixed. Then makes ten mutants, each in its own copy, and reruns the tests on each. It stops at 2 or fewer survivors, after 5 laps, or when two laps in a row make no progress.

3. 03

   Catch what slipped pastAI

   Haiku gets each survivor's exact change and writes a new test file.

4. 04

   Lap by lapScript

   Prints which mutant each lap caught.

A test that fails on the real file would "catch" every mutant, so it can't count. The mutants are limits moved by one, a >= turned into >, roundings flipped, a dropped floor at zero and a dropped multiplication. Roscoe's AI step doesn't check its own reply, so the count picks the route. Survivors go back to Haiku as Stryker's report shows them, which is MuTAP's idea, from 2023.

Two laps on the checkout file#

Recorded runplant-the-bugs-run-v1

Watch the runVideo · 0:27

On lap 1 of the sample run, the script reported "the tests caught 8 of 10 planted bugs (38 tests, but 3 fail on the real code, so only the other 35 count)". The two survivors, Math.round changed to Math.ceil on line 34 and to Math.floor on line 39, went back to Haiku with the three failing tests.

After lap 2, the report opened with "Planted bugs caught: 10 of 10, up from 8 on the first lap".

Lap 1's green tests would have let both rounding slips through. Break a copy of the code before you trust the tests.

“Break a copy of the code before you trust the tests.”

Aiming at the survivors is fair#

Telling the AI which mutants survived looks like teaching to the test, but the tools work this way. Meta's ACH has a model write tests aimed at bugs the current tests miss, and in Messenger and WhatsApp test-a-thons engineers accepted 73% of them, by Meta's count. A test that notices a coupon rounded down is a real rounding test, whoever it was aimed at. Roscoe's own bug-hunt benchmark settles whether a mutant is really a bug the same way, by running it: a reference test must pass on the clean code and fail on the changed code.

What the count can't tell you#

The count says the tests would notice a change, and nothing about whether the code is right. AI-written tests tend to assert what the code does rather than what it should do, so if your file already has a bug, the tests will likely expect it. A 2026 replication found the share of mutants caught stops being a reliable signal when the code under test may already be buggy.

Ten mutants measure those ten, and the rewrite saw which ones survived. The demo's mutant maker is a few string rules for one self-contained JavaScript or TypeScript file; for a whole repo, run Stryker. And lap 1's 8 of 10 comes from one run on the sample, a single scenario file, so it predicts nothing about other files.

A single Claude Code session can gate on a script too: a Stop hook blocks the turn from ending until your check passes, though Anthropic's guide says Claude Code ends the turn after 8 blocks in a row.

Try it on your own file#

Try the scenario file first, with Claude Code or Cowork driving, to see each lap's count, the table of which mutant each lap caught and the final test file, under a $1 spending cap.

Swap in your own file and this is your workflow for checking what an AI's tests actually catch.