AI generated unit tests raise coverage. Mutation testing shows what they miss.
AI generated unit tests match human coverage, research shows, but coverage is not fault detection. How to evaluate AI tests with mutation testing.
If your team, or your vendor, now uses coding agents, a growing share of your test suite is being written by them. AI generated unit tests look impressive in a pull request: long, full of assertions, and good for the coverage number. Recent research confirms they do raise coverage about as well as human-written tests. It also shows why coverage is the wrong way to judge them — and what to use instead.
How much of your test code AI now writes
A study presented at MSR 2026, "Testing with AI Agents", analysed 2,232 commits containing test-related changes from the AIDev dataset of real-world repositories. Its findings:
- AI agents authored 16.4% of all commits that added tests.
- AI-generated test methods have a distinct shape: longer code and a higher density of assertions, with lower cyclomatic complexity because they follow linear logic.
- AI-generated tests contribute to code coverage comparably to human-written tests, frequently producing positive coverage gains.
On the metric most teams track, then, AI tests are doing fine. That is the problem.
Is code coverage a good measure of test quality?
No. Coverage tells you which lines ran during the tests. It does not tell you whether the tests would notice if those lines were wrong. A test that calls a function and asserts that the result is not null covers every line and catches almost nothing.
The traits the MSR study found — many assertions, simple linear structure — can go either way. Many precise assertions on meaningful outputs are good. Many assertions that restate the implementation, check trivial properties or mirror mocks are noise that inflates confidence. You cannot tell which from coverage.
What is mutation testing?
Mutation testing measures whether tests catch bugs by introducing small, deliberate bugs — mutants — into the code, such as flipping a comparison or changing a constant, and running the suite against each. A mutant the tests fail on is "killed"; one they pass is a blind spot. The share of mutants killed is the mutation score. Tools such as PIT for Java do this automatically.
Research on AI test generation shows why it matters. A February 2026 paper, "Test vs Mutant", notes that most work on LLM-generated tests has focused on coverage and readability, with little attention to how robustly the tests detect bugs. Its AdverTest system pits a test-writing agent against a mutant-generating agent that keeps probing the suite's blind spots. On the Defects4J benchmark it improved fault detection by 8.56% over the best existing LLM-based methods and by 63.3% over the classic generator EvoSuite, while also improving line and branch coverage. The gain came from optimising for killed mutants, not for coverage.
How to evaluate AI generated unit tests
A practical acceptance standard for AI-written tests, whether they come from your own agents or a vendor's:
- Measure mutation score on changed code. Run mutation testing on the modules a pull request touches, not the whole codebase. It keeps run times manageable and focuses on new tests.
- Set a floor, then raise it. Start with the score your best human-written modules achieve today, and require new tests to meet it.
- Read the surviving mutants. Each survivor is a concrete missing test. Feeding survivors back to the agent that wrote the tests is a simple version of the adversarial loop in the research.
- Check assertions, not just counts. Reject tests whose assertions only restate the implementation or check that something exists.
- Test behaviour under failure. Linear, happy-path tests are the pattern the MSR study observed. Require tests for errors, empty inputs, timeouts and boundaries.
- Keep humans on the specification. Tests generated from the same requirement as the code will agree with a wrong requirement, as we showed in our piece on acceptance criteria review.
Can AI write good unit tests?
Yes, when it is held to the right bar. AI agents are well suited to the tedious parts of testing — enumerating cases, writing fixtures, covering boundaries a tired developer skips — and the research suggests they can be pushed toward real fault detection when mutation feedback is in the loop. Left to optimise for coverage, they produce suites that look thorough and miss bugs.
Two related habits help. Run agent tasks more than once and compare, because agents are inconsistent in ways a single run hides — see AI agent consistency testing. And on legacy code, capture current behaviour before agents refactor anything, with characterization tests.
How do you review AI generated tests in a pull request?
- Look first at what is asserted, then at how much.
- Ask what bug each test would catch. If nobody can say, the test is probably padding.
- Check that mocks are not so broad that the test only verifies the mock.
- Look at the mutation report for the changed files before approving.
Coverage will keep rising as agents write more of your tests. The number that tells you whether you are safer is the mutation score. If you are receiving AI-generated test suites and are not sure what they are worth, our QA and automation team can add mutation testing to your CI and set an acceptance standard your vendors must meet.
Frequently asked questions
AI can write good unit tests when held to a fault-detection standard rather than coverage alone. Research on adversarial test generation improved fault detection by 8.56% over leading LLM methods by feeding surviving mutants back to the test-writing agent.
Code coverage is not a good measure of test quality on its own. It shows which lines ran during tests, not whether the tests would fail if those lines were wrong. Mutation testing measures the second question directly.
Mutation testing introduces small deliberate bugs, called mutants, into code and runs the test suite against each one. Tests that fail on a mutant have killed it; the percentage of killed mutants is the mutation score, a direct measure of bug-detection ability.
Review AI generated tests by checking what each test asserts and which bug it would catch, confirming mocks do not hide the behaviour under test, requiring failure and boundary cases, and checking the mutation score for changed files before approval.
An MSR 2026 study of 2,232 test-related commits in real-world repositories found that AI agents authored 16.4% of the commits adding tests, with longer, assertion-dense test methods and coverage gains comparable to human-written tests.
