Mutation testing AI generated tests: what coverage hides

Mutation testing AI generated tests is the cheapest way to find out whether a suite that reports 90% coverage would actually notice a bug. Coverage records that a line ran. Mutation testing changes the line and checks whether a test complains. A generated suite can score well on the first and badly on the second, and here is how to tell.

The number everyone quotes is a correlation coefficient

Before anything else, a correction. A figure doing the rounds in 2026 posts about AI test quality is "0.546 for LLM-generated tests against 0.690 for human-written" presented as mutation scores. It is not. Those numbers come from a replicability study by Junda Zhao, Shurui Zhou and Eldan Cohen (July 2026), and they are Pearson correlation coefficients measuring how well mutation score predicts real-bug detection — not scores that any suite achieved. The study covers 11 LLMs, 8,268 generated suites and 101,123 test cases against Defects4J v3.0.

Its actual finding is more interesting than the misquote, and more inconvenient for everyone selling mutation testing: the usefulness of coverage and mutation is highly context-dependent. Where the code under test can reasonably be assumed correct, both give meaningful signal for comparing generators. Where the code under test may itself contain bugs, coverage becomes unreliable and mutation analysis is inapplicable — because mutation measures your tests against the code as written, and if the code is already wrong, killing a mutant of it proves nothing about the bug.

That is the honest frame. Mutation testing tells you whether your tests pin the behaviour your code currently has. It does not tell you whether that behaviour is correct, which is a different review, and one we have written about separately in the case where six tests passed and the spec was the bug.

Three failure modes you can find by reading, before you run anything

You do not need a metric to catch the common ones. Each of these executes the code under test, so each one earns full line coverage, and none of them would fail if the function returned something else entirely.

Calling without asserting. The test runs the function, the function does not throw, the test passes:

@Test
void calculatesDiscount() {
    pricing.discountFor(order, customer);
}

Covered lines: all of them. Behaviour pinned: none. Change discountFor to return zero and this test stays green.

Asserting existence where a value is specified. The requirement says a gold customer on a 1,000 AED order gets 150 AED off. The test says the result exists:

@Test
void goldCustomerGetsDiscount() {
    var result = pricing.discountFor(order, goldCustomer);
    assertNotNull(result);
}

Every mutant that changes the arithmetic survives this. So does swapping the gold and silver branches. assertNotNull is the right assertion exactly once — when "not null" is the specification.

The tautology in the exception handler. This one is almost always generated rather than written:

try {
    pricing.discountFor(null, customer);
    fail("expected an exception");
} catch (Exception e) {
    assertTrue(true);
}

The catch block asserts nothing. It will pass on a NullPointerException from an unrelated line, which is the usual case and the opposite of what was intended. The fix is to assert the type and the message, or to use assertThrows and assert on what it returns.

A ten-minute grep across a generated suite for assertNotNull, assertTrue(true) and test methods containing no assertion at all will usually find more than a first mutation run does, and costs nothing in CI time. Do that first.

What mutation testing AI generated tests actually measures

A mutation testing tool makes small single changes to your source — flips a boundary, negates a condition, replaces a return value with a default, removes a call. Each change is a mutant. It runs the tests that cover the changed line. If a test fails, the mutant is killed. If every test still passes, the mutant survived, and you now have a precise, reproducible statement of a change your suite would not notice. The mutation score is the proportion killed.

The gap this exposes can be enormous. In the experiments behind MutGen (Guancheng Wang, Qinghua Xu, Lionel Briand and Kui Liu, August 2025), LLM-generated tests for one subject from the HumanEval-Java benchmark reached 100% line and branch coverage with a mutation score of 4%. That is a single subject rather than a portfolio average, and it should be read as an existence proof rather than a typical result — but as an existence proof it is decisive. A coverage number and a test suite's fault-detection ability are not the same quantity, and on that subject they were almost unrelated.

A survived mutant is also a better bug report than a coverage gap. "Line 47 is uncovered" tells you to write a test. "Changing >= to > on line 47 breaks nothing" tells you which test to write and what it must assert.

The strongest case against doing any of this

Two arguments, both serious.

The first is that the premise may be out of date. Douglas Leith's study of Claude-authored Python tests (August 2026) compared AI-written tests against the human-written corpora of Django and Pandas using three fault-injection protocols and a seven-axis design rubric, and concluded that tests from recent Claude models were no weaker than either human corpus. If your generator is current and your prompting is good, the quality gap this post assumes may not be your gap. Measure before you believe anyone, us included.

The second is cost. Mutation testing runs your test suite once per mutant, over thousands of mutants. Naively applied to a repository of any size, it does not slow your pipeline down — it stops it being a pipeline. Teams that adopt it per-commit abandon it within a month, which is the real reason it has stayed niche since the 1970s. If you cannot solve the cost problem, do not start.

How to put a mutation run in CI without making it unusable

Three constraints, in order of how much time they save.

Scope to what changed. This is most of the win. For StrykerJS, derive the file list from git and pass it: npx stryker run --incremental --mutate $(git diff --name-only origin/main...HEAD -- 'src/**/*.ts' | tr '\n' ','). The incremental mode docs are worth reading closely before you trust the result: Stryker reuses a prior verdict when a mutant was killed and the culprit test is unchanged, it does a git-like diff only of mutated files and test files, and it explicitly will not detect changes in anything else — dependency upgrades, environment variables, snapshot files. The dry run still happens every time, so there is a floor under the cost. For PIT, the equivalent is incremental analysis via withHistory, plus narrowing targetClasses. PIT's own documentation is admirably blunt that it tracks changes to superclasses and outer classes but not broader dependencies, and that the assumption this is safe "is currently unproven".

Run it nightly, not per-commit. A full run on the default branch overnight gives you a trend and a backlog of surviving mutants to work through, without any developer waiting. Keep the changed-files run in the pull request if it fits your budget, and fail nothing on it at first — report only.

Set a threshold that fails the build, eventually. StrykerJS ships thresholds of { high: 80, low: 60, break: null }, where break is null by default and a score below it exits with code 1. That default is the right starting point: measure for a few weeks, find your real baseline, then set break a little below it so the build fails on regression rather than on the absolute number. A threshold set to an aspirational figure on day one gets commented out on day three.

The same reasoning about pipeline budget that we applied in speeding up CI runs under AI-era load applies here: the question is never whether a check is valuable, it is what you are willing to spend per commit and what moves to nightly.

Which surviving mutants are worth ignoring

Chasing a mutation score to 100% is a mistake, and the tooling authors say so. Some mutants are equivalent — semantically identical to the original, so no test can possibly kill them. The MutGen paper notes that PIT includes mechanisms to reduce their creation but that the remainder still prevent a 100% score. Budget for them rather than fighting them.

The categories we would exclude or accept as survivors:

• Logging and telemetry. Mutating a log message is not a fault worth a test. Exclude these packages.
• Defensive checks that cannot be reached. A null guard on a parameter the type system already prevents is unkillable by construction.
• Performance-only code. A cache that is removed by a mutant and still returns the right answer is correctly not killed. The mutant is telling you the cache is not part of the contract, which is true.
• Generated and boilerplate code. Equals, hashCode, builders, DTO mappers. High mutant counts, near-zero signal.
• Timestamps, randomness and identifiers. Killing these reliably usually means asserting on an implementation detail you will regret.

Write the exclusions into configuration with a comment giving the reason, and review the list quarterly. An exclusion nobody can justify is how a mutation score becomes as hollow as the coverage number it replaced. The same discipline we apply to flaky test metrics applies: a number that is managed rather than earned stops being a signal.

Feeding surviving mutants back to the generator

This is the part that turns a metric into a workflow, and it has been demonstrated at both research and industrial scale.

MutGen puts the mutation result into the prompt: live and uncovered mutant information is included in prompt construction, so the model is asked for a test that kills a named surviving mutant rather than for "more tests". On HumanEval-Java it reached a mutation score of 89.5% against EvoSuite's 69.5%, and on Leetcode-Java 89.1% against 58.9%. The practical detail worth stealing is the stopping rule: the authors cap repetitions at four, having observed that scores converge after the fourth iteration. If your loop is still running on iteration seven, it is burning tokens.

Meta's ACH system, described in Mutation-Guided LLM-based Test Generation at Meta (Foster et al., January 2025), inverts the usual proportions: it generates relatively few mutants, targeted at a specific class of concern, rather than exhaustively. Across 10,795 Android Kotlin classes it produced 9,095 mutants and 571 privacy-hardening test cases, of which 73% were accepted by Messenger and WhatsApp engineers and 36% judged privacy-relevant. The lesson is that mutation guidance works best pointed at a concern you can name, not at the whole codebase.

The loop we would run on a generated suite: generate, run coverage to find what was never executed, run mutation on the changed files to find what was executed but unasserted, then feed the surviving mutants back as explicit constraints — "write a test that fails when line 47's >= becomes >" — and cap it at four rounds. What survives after that goes to a human, because it is either an equivalent mutant or a genuine gap, and only a person can tell you which.

None of this replaces the generation step; it is the acceptance test for it. If you are still deciding how to generate the tests in the first place, that is the subject of our earlier post on how AI generated unit tests raise coverage and what mutation testing shows they miss. This post is about what to do with the output. The decision it should change is small and concrete: before you accept a generated suite into your regression set, run mutation on the diff, and treat the surviving mutants as the review comments. If that is work you would rather hand to someone, our QA and automation practice does exactly this wiring.