AI can write a thousand tests before lunch. That is the problem, not the win. A tool that generates tests fast also generates flake fast, writes assertions that check nothing, and mails your team a maintenance bill six months later. The question was never “can it write tests.” Every tool can. The question is “can I trust these tests in CI without babysitting them,” and that has a measurable answer. This kit is how you measure it, and how you treat the decision as a signal problem instead of a coverage race.
What happens by default: you point the tool at the repo, coverage jumps 20 points, and you ship it. Nobody checks whether the new tests would fail if the code broke. Flake gets muted or retried until green. Six months later the suite is slower, noisier, and no better at catching a regression, and your team spends its week triaging failures that were never real.
What good evaluation does: it accepts a generated test only when it kills mutants and stays under the flake gate. Coverage becomes a side effect, not the goal. You judge the tool on mutation score, flake rate, and escaped defects, run a pilot on your own code, and let the numbers, not a vendor slide, make the call.
Line coverage tells you which lines ran, not whether the test would notice if they broke. A generator can execute a payment function, assert nothing, and push coverage from 61% to 88%. Score whether new coverage lands on changed, risky, or business-critical paths, or just on easy getters added for the dashboard. Relevant coverage counts. Vanity coverage does not.
This is the lens that costs you money later, so weight it heaviest. Measure it with mutation score: inject small faults (flip a comparison, drop a boundary, null a return) and count how many the generated tests catch. Assertion-free execution and tautological assertions both pass coverage and prove nothing. Tests that cannot kill mutants are decoration.
A generated suite you cannot read, cannot heal, and cannot keep deterministic is a liability with good coverage. Score flake introduced per 100 tests over 20 runs, self-healing precision, false-positive rate, and whether a reviewer can own the tests in a diff. A tool that pushes flake up moves cost downstream and hides it.
Do not evaluate on a toy repo. Pick one real service and one real end-to-end flow, freeze the code, and record your current flake rate, risk-path coverage, and triage hours. Seed 8 to 12 known defects into a branch the tool cannot see. Gate 1: the tool integrates without prod-data access and without auto-merge, or you stop.
Generate tests for the frozen service and run them against the seeded-defect branch. Measure mutation score on changed code and count escaped defects. Have two engineers review 30 random tests against the rubric. Gate 2: mutation score at or above 60% and zero seeded criticals escaped, or you stop.
Run the suite 20 times on an unchanged commit to compute flake and false-positive rate. Make three realistic UI or API changes and measure self-healing precision. Gate 3: flake under 3%, false positives under 15%, self-heal precision at or above 80%, or you stop.
Have the team fix, extend, and refactor the generated tests for a week to get a real maintenance ratio. Complete the test-data security review. Fill in the weighted scorecard from evidence. Gate 4: scorecard at or above 70%, no criterion scoring a 1, security review passed. That is your buy, conditional, or no.
Add AI test generation to a disciplined suite and you get more good tests, faster. Add it to a suite that already flakes and you get more noise, faster, with a confident label on it. The coverage number will go up either way, which is exactly why coverage is the wrong thing to trust. Mutation score and flake rate tell you the truth. Run the pilot, hold the gates, and let the tool earn its place instead of assuming it.
Not on its own. Coverage measures which lines ran, not whether a test would catch a bug in them. Run mutation testing on the same suite. If the mutation score is low, those extra points are decoration, and the suite is now slower and flakier for no added signal.
On boilerplate and breadth: filling gaps on changed code, drafting edge cases a human would skip, and taking the first pass at a large surface. It helps most when a human reviews with the trust-but-verify rubric before anything merges. The generation is the easy part. The gate is the product.
QA leads and directors of QA evaluating an AI or agentic test-generation tool, plus the engineering managers who will own the CI bill and the reviewers who accept or reject the tests.
Because they are what cost you money after the purchase. A missed bug reaches production. A flaky test burns roughly 3.7 engineer-hours per fix and trains the team to ignore red. Coverage optics cost nothing to inflate, so they earn a low weight on purpose.
Four weeks on a real service, one gate per week. Week 1 baseline and setup, week 2 signal, week 3 stability under CI, week 4 maintainability and the decision. Miss a gate and you stop, you do not “give it another week.” Momentum is how bad tools get bought.
