AI can write a thousand tests before lunch. That is the problem, not the win. A tool that generates tests fast also generates flake fast, writes assertions that check nothing, and mails your team a maintenance bill six months later. The question was never “can it write tests.” Every tool can. The question is “can I trust these tests in CI without babysitting them,” and that has a measurable answer. This kit is how you measure it, and how you treat the decision as a signal problem instead of a coverage race.
Does the New Coverage Touch Anything Risky
Line coverage tells you which lines ran, not whether the test would notice if they broke. A generator can execute a payment function, assert nothing, and push coverage from 61% to 88%. Score whether new coverage lands on changed, risky, or business-critical paths, or just on easy getters added for the dashboard. Relevant coverage counts. Vanity coverage does not.
Would the Test Fail for the Right Reason
This is the lens that costs you money later, so weight it heaviest. Measure it with mutation score: inject small faults (flip a comparison, drop a boundary, null a return) and count how many the generated tests catch. Assertion-free execution and tautological assertions both pass coverage and prove nothing. Tests that cannot kill mutants are decoration.
Can a Human Own This in CI
A generated suite you cannot read, cannot heal, and cannot keep deterministic is a liability with good coverage. Score flake introduced per 100 tests over 20 runs, self-healing precision, false-positive rate, and whether a reviewer can own the tests in a diff. A tool that pushes flake up moves cost downstream and hides it.
Do not evaluate on a toy repo. Pick one real service and one real end-to-end flow, freeze the code, and record your current flake rate, risk-path coverage, and triage hours. Seed 8 to 12 known defects into a branch the tool cannot see. Gate 1: the tool integrates without prod-data access and without auto-merge, or you stop.
Generate tests for the frozen service and run them against the seeded-defect branch. Measure mutation score on changed code and count escaped defects. Have two engineers review 30 random tests against the rubric. Gate 2: mutation score at or above 60% and zero seeded criticals escaped, or you stop.
Run the suite 20 times on an unchanged commit to compute flake and false-positive rate. Make three realistic UI or API changes and measure self-healing precision. Gate 3: flake under 3%, false positives under 15%, self-heal precision at or above 80%, or you stop.
Have the team fix, extend, and refactor the generated tests for a week to get a real maintenance ratio. Complete the test-data security review. Fill in the weighted scorecard from evidence. Gate 4: scorecard at or above 70%, no criterion scoring a 1, security review passed. That is your buy, conditional, or no.
Not on its own. Coverage measures which lines ran, not whether a test would catch a bug in them. Run mutation testing on the same suite. If the mutation score is low, those extra points are decoration, and the suite is now slower and flakier for no added signal.
Because they are what cost you money after the purchase. A missed bug reaches production. A flaky test burns roughly 3.7 engineer-hours per fix and trains the team to ignore red. Coverage optics cost nothing to inflate, so they earn a low weight on purpose.
On boilerplate and breadth: filling gaps on changed code, drafting edge cases a human would skip, and taking the first pass at a large surface. It helps most when a human reviews with the trust-but-verify rubric before anything merges. The generation is the easy part. The gate is the product.
Four weeks on a real service, one gate per week. Week 1 baseline and setup, week 2 signal, week 3 stability under CI, week 4 maintainability and the decision. Miss a gate and you stop, you do not “give it another week.” Momentum is how bad tools get bought.
QA leads and directors of QA evaluating an AI or agentic test-generation tool, plus the engineering managers who will own the CI bill and the reviewers who accept or reject the tests.
Drop your details and we'll send AI Test Generation Evaluation Kit straight to your inbox - no spam, unsubscribe anytime.
Talk through how this applies to your roadmap with our engineering leads - a working session, not a sales pitch.
Download White Paper