Ask an AI the same question twice and you can get two answers. Ship the same code for months and watch quality drift because the model changed under you. Run your tests and get a green light that means far less than it used to. This guide lays out a QE methodology built for AI: evals, baselines, and drift alarms.
What most teams still rely on: exact-match tests, binary pass/fail, and a one-time launch sign-off all of which assume a determinism AI doesn't have, so they go flaky, hide graded failures, and certify a moment instead of a living product.
What quality engineering for AI looks like: evals that score outputs across a representative set, baselines that make normal variance manageable, and drift alarms that catch the silent regressions before your customers do.
Score outputs against criteria accuracy, relevance, completeness, safety across a representative set of inputs, rather than checking against one canonical string. The unit of testing becomes "how good, over a distribution," not "did it equal X."
Establish the current measured quality and test every change against it. Did this prompt, model, or retrieval tweak move the numbers up or down? Baselines turn a noisy system into something you can reason about statistically.
Run your eval set continuously in production and alarm when quality drifts. Because a hosted model can change under you, evaluation is a monitor, not a launch gate you pass once.
Public benchmarks are contaminated and generic; a model can ace them and fail your use case. Draw your eval set from the real inputs your system will face.
Include the hallucination-prone, ambiguous, and adversarial cases, and the ones where the safe answer is "I don't know." That's where AI products actually fail.
Accuracy is rarely the only axis. Score groundedness, safety, tone, format, refusal behavior, and cost as your product requires.
Every new failure you find in production becomes a new eval case, so the same problem can never silently return. The eval set is an asset that compounds.
No. Studies show "deterministic" settings still vary due to floating-point and batching effects, and models give different answers to the same prompt even at temperature zero a meaningful share of the time. You need evals and tolerances, not exact-match assertions.
A scored assessment of model output against criteria accuracy, relevance, safety, format run across a representative set of inputs, rather than a check against one correct string. It's the AI-era replacement for the unit test as your primary quality signal.
Because hosted models change. A Stanford/Berkeley study documented GPT-4's accuracy on a task dropping from 84% to 51% within months. Without continuous evals and drift alarms, that kind of regression reaches you through your users.
No. Popular benchmarks are contaminated one analysis found ~29% of MMLU items showed contamination signs so public scores overstate real-world quality. Build an eval set from your own data.
Heads of AI, Directors of QA, and quality engineering leaders shipping AI products that need to stay reliable in production.
Drop your details and we'll send Testing Non-Deterministic Systems straight to your inbox - no spam, unsubscribe anytime.
Talk through how this applies to your roadmap with our engineering leads - a working session, not a sales pitch.
Download White Paper