Point an agent at your app and it explores, writes tests, and finds bugs while you sleep — that's the pitch. The reality is more useful: genuinely capable, genuinely limited, in ways the demos don't show. This field report separates what autonomous testing delivers today from the vendor deck, and hands you the criteria to evaluate any tool.
The vendor deck: quote a high benchmark score, imply hands-off autonomy, and let you assume the agent's quality on curated demos matches its quality on your unfamiliar, domain-specific codebase.
The field reality: autonomous generation plus human supervision beats either alone — most raw agent output gets filtered out, and the value comes from grounding the agent in intent and keeping a human in the acceptance loop.
Agents are good at producing many candidate tests fast — unit tests, edge cases, boundary conditions a human might not enumerate — as a way to bootstrap coverage on under-tested code.
An autonomous agent can navigate an app and surface unexpected states and obvious breakages — a tireless, if shallow, exploratory tester that finds the paths nobody tried under time pressure.
When a selector or minor UI detail changes, the agent updates the test instead of failing it — attacking one of QA's most tedious and expensive cost centers.
How does the tool know what correct behavior is? Does it consume specs, requirements, or existing intent — or does it just assert current behavior and risk locking in bugs?
What fraction of generated tests build, pass reliably, and add real coverage? Compare against Meta's public baseline of ~75% built / 57% passed / 25% coverage — and check whether the noise lands on your team.
Does the tool reduce flakiness or add to it? Can it self-heal — and how often does self-healing quietly hide a real regression? Measure the net maintenance effect after adoption, not in the demo.
Where does a human review and accept the agent's output, and can you audit what it tested and concluded? The successful deployments work because a person approves what the agent proposes.
Agentic testing is neither the revolution the vendors sell nor the gimmick the skeptics dismiss. The models are genuinely capable, the deployments are useful with heavy filtering and human acceptance, and the limits are real on novel problems. Deploy where the risk is low and the toil is high, keep a human owning acceptance, and measure by escaped defects — not by tests produced.
On curated benchmarks of real issues, yes — leading models score around 77% on SWE-bench Verified. On novel problems the model hasn't effectively seen, performance drops substantially, so "on its own" holds far better in the demo than on your codebase
Trusting generated tests that assert the current (wrong) behavior is correct, and drowning in flaky, low-value tests that erode confidence in the whole suite. Both come from deploying without a filter and without human acceptance.
Directors of QA, Heads of Quality Engineering, and test leaders deciding where autonomous testing fits their stack and where it doesn't.
No. The most successful real deployment worked because engineers filtered and accepted the output — most raw generated tests were discarded. Agentic testing shifts QA work toward supervision, grounding, and judgment; it doesn't remove the need for it.
Use the four criteria — correctness grounding, signal-to-noise versus a public baseline, effect on flakiness and maintenance, and where a human stays in control. Trial it on your own codebase; don't buy on benchmark claims.
