# Would your AI-written tests catch the next bug?

Say an AI wrote thorough tests for a shop's checkout math, and they all pass. Later a teammate's refactor, or an AI agent's edit, makes a bulk discount round a cent high, and those tests are meant to catch it. In one real run on the sample, a scenario file, the AI's first try had 38 tests, and the 35 that passed on the real code caught 8 of 10 small deliberate changes like that one.

Both misses were roundings: a bulk-discounted line price rounded up instead of to the nearest cent, and a percent coupon rounded down. All 35 tests were green, and green says little. Coverage only records which lines ran, so [a test with no assertions still counts](https://pitest.org/), and for AI-written suites [a 2026 study](https://arxiv.org/abs/2609.24341) found coverage near its ceiling.

To find out whether tests would notice, a tool makes one small deliberate change to a copy of your code, the kind of slip a later edit could make, such as a `>=` turned into `>`, and reruns your tests. Each such change is [a mutant](https://stryker-mutator.io/docs/), and one that survives means [you're missing a test](https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/). That's mutation testing, [first proposed in 1971](https://en.wikipedia.org/wiki/Mutation_testing). [Anthropic's Claude Code guide](https://www.anthropic.com/engineering/claude-code-best-practices) says the agent doing the work shouldn't be the one grading it, so when an AI writes your tests, let a script's count of mutants decide when they're done, not the model.

```plainwords
term: Mutant
means: One small deliberate change to a copy of your code, such as a `>=` turned into `>`. If a test fails, it's caught; if every test still passes, it survived.
```

## How teams use it

[Stryker](https://stryker-mutator.io/) is a free, open-source mutation testing tool for JavaScript and TypeScript, C# and Scala, and [PIT](https://pitest.org/) is its Java counterpart. Stryker works in [a sandbox copy](https://stryker-mutator.io/docs/stryker-js/configuration/) of your project, so your real files stay as they are. Each surviving mutant in its report points at a line whose tests would miss a change, so you add or tighten a test there. PIT advises running it often on just the changed code, including on pull requests, and [Stryker can too](https://stryker-mutator.io/docs/stryker-js/incremental/), for faster CI runs. [At Google](https://research.google/pubs/state-of-mutation-testing-at-google/), 6,000 engineers used it on the code changes they wrote or reviewed, as of 2018.

Mutants stand in for real bugs well. [A 2014 study of 357 real faults](https://homes.cs.washington.edu/~mernst/pubs/mutation-effectiveness-fse2014-abstract.html) in 5 open-source Java programs found that catching mutants is significantly correlated with catching real ones, independently of coverage.

Reading the tests is no substitute. Two comments in Roscoe's own code record tests that stayed green with the code they covered deleted: in one, every assertion off Windows compared `false` with `false`. Nor is asking for thoroughness, since the run's first prompt already used the word.

## Six steps and a loop

```workflow
id: demo-12-plant-the-bugs
caption: The AI writes tests, a script makes ten mutants and counts the misses, and the AI rewrites until 2 or fewer survive.
```

The demo calls its mutants planted bugs.

```steps
step: Write the tests
kind: ai
text: Haiku gets the file and a plain ask for thorough tests, with no word about mutants.

step: How many planted bugs slip past?
kind: script
text: Runs the tests on the real file, where a failing test doesn't count and goes back to be fixed. Then makes ten mutants, each in its own copy, and reruns the tests on each. It stops at 2 or fewer survivors, after 5 laps, or when two laps in a row make no progress.

step: Catch what slipped past
kind: ai
text: Haiku gets each survivor's exact change and writes a new test file.

step: Lap by lap
kind: script
text: Prints which mutant each lap caught.
```

A test that fails on the real file would "catch" every mutant, so it can't count. The mutants are limits moved by one, a `>=` turned into `>`, roundings flipped, a dropped floor at zero and a dropped multiplication. Roscoe's AI step [doesn't check its own reply](/docs/nodes/ai-agent), so the count picks the route. Survivors go back to Haiku as [Stryker's report](https://stryker-mutator.io/docs/) shows them, which is [MuTAP's idea](https://arxiv.org/abs/2308.16557), from 2023.

## Two laps on the checkout file

```video
key: media/blog/plant-the-bugs-run-v1
poster: plant-the-bugs-run
duration: 27.43
caption: The scenario run, where lap 1 catches 8 of 10 mutants and lap 2 catches all 10.
```

On lap 1 of the sample run, the script reported "the tests caught 8 of 10 planted bugs (38 tests, but 3 fail on the real code, so only the other 35 count)". The two survivors, `Math.round` changed to `Math.ceil` on line 34 and to `Math.floor` on line 39, went back to Haiku with the three failing tests.

After lap 2, the report opened with "Planted bugs caught: 10 of 10, up from 8 on the first lap".

```table
label: Lap by lap
note: Sample run
meter: Mutants caught
head: Lap | Tests that count | Mutants caught | Slipped past
row: 1 | 35 of 38 | 8 of 10 | `Math.round → Math.ceil` line 34; `Math.round → Math.floor` line 39
row: 2 | 40 of 40 | 10 of 10 | None
caption: The run on the sample scenario file, from the counts the script printed each lap.
```

```runstrip
time: 301 s
cost: $0.24
cap: $1
model: Haiku
caption: The sample run. Cost is a subscription-equivalent estimate at list price, not a bill.
```

Lap 1's green tests would have let both rounding slips through. Break a copy of the code before you trust the tests.

```pullquote
text: Break a copy of the code before you trust the tests.
```

## Aiming at the survivors is fair

Telling the AI which mutants survived looks like teaching to the test, but the tools work this way. [Meta's ACH](https://arxiv.org/abs/2501.12862) has a model write tests aimed at bugs the current tests miss, and in Messenger and WhatsApp test-a-thons engineers accepted 73% of them, by Meta's count. A test that notices a coupon rounded down is a real rounding test, whoever it was aimed at. Roscoe's own bug-hunt benchmark settles whether a mutant is really a bug the same way, by running it: a reference test must pass on the clean code and fail on the changed code.

## What the count can't tell you

The count says the tests would notice a change, and nothing about whether the code is right. AI-written tests tend to [assert what the code does rather than what it should do](https://arxiv.org/abs/2410.21136), so if your file already has a bug, the tests will likely expect it. [A 2026 replication](https://arxiv.org/abs/2607.22880) found the share of mutants caught stops being a reliable signal when the code under test may already be buggy.

Ten mutants measure those ten, and the rewrite saw which ones survived. The demo's mutant maker is a few string rules for one self-contained JavaScript or TypeScript file; for a whole repo, run Stryker. And lap 1's 8 of 10 comes from one run on the sample, a single scenario file, so it predicts nothing about other files.

A single Claude Code session can gate on a script too: a Stop hook blocks the turn from ending until your check passes, though [Anthropic's guide says](https://www.anthropic.com/engineering/claude-code-best-practices) Claude Code ends the turn after 8 blocks in a row.

## Try it on your own file

Try the scenario file first, with Claude Code or Cowork driving, to see each lap's count, the table of which mutant each lap caught and the final test file, under a $1 spending cap.

Swap in your own file and this is your workflow for checking what an AI's tests actually catch.

```download
workflow: demo-12-plant-the-bugs
```