Point an agent at your app and it explores, writes tests, and finds bugs while you sleep that's the pitch. The reality is more useful: genuinely capable, genuinely limited, in ways the demos don't show. This field report separates what autonomous testing delivers today from the vendor deck, and hands you the criteria to evaluate any tool.
The vendor deck: quote a high benchmark score, imply hands-off autonomy, and let you assume the agent's quality on curated demos matches its quality on your unfamiliar, domain-specific codebase.
The field reality: autonomous generation plus human supervision beats either alone most raw agent output gets filtered out, and the value comes from grounding the agent in intent and keeping a human in the acceptance loop.
Agents are good at producing many candidate tests fast unit tests, edge cases, boundary conditions a human might not enumerate as a way to bootstrap coverage on under-tested code.
An autonomous agent can navigate an app and surface unexpected states and obvious breakages a tireless, if shallow, exploratory tester that finds the paths nobody tried under time pressure.
When a selector or minor UI detail changes, the agent updates the test instead of failing it attacking one of QA's most tedious and expensive cost centers.
How does the tool know what correct behavior is? Does it consume specs, requirements, or existing intent or does it just assert current behavior and risk locking in bugs?
What fraction of generated tests build, pass reliably, and add real coverage? Compare against Meta's public baseline of ~75% built / 57% passed / 25% coverage and check whether the noise lands on your team.
Does the tool reduce flakiness or add to it? Can it self-heal and how often does self-healing quietly hide a real regression? Measure the net maintenance effect after adoption, not in the demo.
Where does a human review and accept the agent's output, and can you audit what it tested and concluded? The successful deployments work because a person approves what the agent proposes.
On curated benchmarks of real issues, yes — leading models score around 77% on SWE-bench Verified. On novel problems the model hasn't effectively seen, performance drops substantially, so "on its own" holds far better in the demo than on your codebase
No. The most successful real deployment worked because engineers filtered and accepted the output — most raw generated tests were discarded. Agentic testing shifts QA work toward supervision, grounding, and judgment; it doesn't remove the need for it.
Trusting generated tests that assert the current (wrong) behavior is correct, and drowning in flaky, low-value tests that erode confidence in the whole suite. Both come from deploying without a filter and without human acceptance.
Use the four criteria — correctness grounding, signal-to-noise versus a public baseline, effect on flakiness and maintenance, and where a human stays in control. Trial it on your own codebase; don't buy on benchmark claims.
Directors of QA, Heads of Quality Engineering, and test leaders deciding where autonomous testing fits their stack and where it doesn't.
Drop your details and we'll send Agentic Testing: Field Report straight to your inbox - no spam, unsubscribe anytime.
Talk through how this applies to your roadmap with our engineering leads - a working session, not a sales pitch.
Download White Paper