An AI evaluation harness in production is the test suite for AI systems. It is a defined set of inputs, expected behavior or quality criteria, and a scoring mechanism that runs on every change. Real examples reveal how teams actually build and use evaluation harnesses, what sizes are practical, which scoring methods work, and how mature evaluation practice differs from teams that ship without systematic evaluation.
The reason evaluation harnesses deserve their own concept traces to the unusual properties of AI systems. Outputs are non-deterministic. Quality is a distribution rather than a binary. There is no exception thrown when the model returns a wrong answer. Traditional unit tests do not work because the same input can produce different outputs across runs. AI evaluation harnesses use statistical scoring against representative inputs to produce signal in this domain that resists traditional testing.
By 2026 evaluation harnesses are recognized as the foundation of reliable AI iteration. Teams that built one early can iterate confidently on prompts, retrieval, and tool design. Teams that did not are guessing whether their changes help. The discipline mirrors traditional software testing in role but differs significantly in mechanics.
The patterns that work share characteristics. Real production cases supplement synthetic test inputs. Multiple scoring methods (exact match, heuristic, judge model) catch different kinds of issues. Continuous integration runs evaluation on every change. Production sampling extends evaluation to current behavior. The harness evolves as new failure modes are discovered. The combination produces working evaluation infrastructure.
This page surveys real evaluation harness implementations across companies. Specific tools evolve quickly; the patterns are more durable than any specific framework choice.
Customer support teams maintain evaluation sets covering common ticket types, edge cases, and known failure cases. Typical sizes range from 200 to 500 cases. Each case includes the input (a customer query), expected behavior (correct answer or escalation), and scoring criteria. The harness runs on every prompt change, model upgrade, or retrieval modification.
The scoring combines methods. Exact match for cases with definitive correct answers. Heuristic checks for structural properties (response includes citation, response is appropriate length, response refuses out-of-scope queries). Judge model evaluation for subjective quality dimensions (response is professional, response is helpful).
RAG systems use Ragas-style metrics for retrieval and generation quality scored separately. Retrieval metrics: did the right chunks come back, what is the precision and recall, what is the mean reciprocal rank. Generation metrics: is the answer faithful to the context, is the answer relevant to the question, is the answer complete. Tools like Ragas automate the metrics; teams build the test sets.
Agent systems evaluate task completion, tool use, and step efficiency. Did the agent complete the task. Did it use the right tools. Did it stay within step and cost budgets. Did it handle errors appropriately. The evaluation captures not just the final output but the path the agent took.
Code generation teams evaluate against test suites for the generated code. Does the code compile. Do tests pass. Does it match style guidelines. Are there security issues. The objective scoring (tests pass or fail) makes coding evaluation more straightforward than evaluation in domains without ground truth.
Start with twenty to fifty representative cases. The mistake is waiting until you have a perfect set of hundreds before running anything. Imperfect coverage that runs is more valuable than perfect coverage that does not.
Pull cases from real production traffic where possible. Logs of past interactions show what the system actually faces. Cherry-pick representative successes, failures, and edge cases. Synthetic cases miss the patterns of real use.
Define scoring criteria explicitly for each case. "Is the response good" is too vague to use. "Does the response correctly identify the customer's product, propose the right next step, and stay under 200 words" is measurable.
Mix scoring methods. Exact match for cases with ground truth. Heuristic checks for structural properties. Judge model evaluation for subjective quality. Each catches different kinds of issues. No single method works for all cases.
Run on every change. Before deploying a prompt update, run the harness. Before adopting a new model version, run the harness. Make running the harness a CI step where possible. The discipline of running on every change is what catches regressions.
Update the harness over time. When you find new failure modes in production, add them as test cases. When use cases evolve, retire test cases that no longer reflect current traffic. The harness is a living artifact that should evolve with the system.
Promptfoo is a popular open-source tool for prompt evaluation. Supports multiple providers, parallel execution, and various scoring methods. Lightweight and easy to integrate. Widely used by teams that want code-based evaluation without managed services.
DeepEval and Ragas focus on RAG-specific evaluation, with metrics for retrieval quality (context relevance, context recall) and generation quality (faithfulness, answer relevance) tuned to retrieval-augmented systems.
Braintrust and LangSmith Evals provide more polished platforms with experiment tracking, comparison views, and integration with their broader observability tools. Useful when teams want managed UX rather than scripts.
Custom evaluation harnesses written in Python or TypeScript work well for many teams. The logic is not complex; what matters is having the harness exist and run. A 200-line script can serve a sophisticated team well.
For LLM-as-judge scoring, the same foundation models used in production can evaluate outputs against rubrics. Some teams use a more capable model as judge than the one being evaluated. Others use the same model. The trade-off is cost versus calibration.
CI integration runs evaluation on every pull request that changes prompts, models, or retrieval configuration. The CI fails the build if quality drops below threshold. The pattern catches regressions before they ship.
Daily evaluation runs against production traffic samples. The scores feed into dashboards. Drops trigger alerts. The pattern catches drift that offline evaluation misses.
A/B comparison evaluation. When considering a model upgrade or significant prompt change, run both versions through the eval set and compare. The comparison shows whether the change helps, hurts, or is neutral on average across the test cases. Detailed comparison shows which specific cases changed.
Regression test additions. When a specific failure case is found in production, add it to the harness as a regression test. Future changes that reintroduce the failure get caught before deployment. Over time the regression test set becomes a record of issues the team has solved.
Threshold-based gates. The eval harness produces scores; CI uses the scores to decide whether to allow deployment. Quality scores must stay above threshold; latency must stay below threshold; cost must stay below threshold. The gates make deployment decisions explicit rather than discretionary.
Minimum useful coverage is 20 to 50 representative cases. Production-grade harnesses for important use cases typically have 100 to 500\. Beyond that, marginal value of additional cases drops, but coverage of edge cases and specific failure modes can justify more.
The right number depends on use case complexity and risk. Simple use cases need fewer cases. Complex use cases or high-stakes systems need more. The 100 to 500 range covers most production scenarios.
Three approaches work. Define expected outputs verbatim where there is a clear correct answer. Define quality criteria as a rubric (the response should accurately identify X, propose Y, avoid Z). Use reference outputs (this is one good answer; others may also be acceptable) with judge model scoring against the reference.
Mix approaches based on case type. Some cases have clear correct answers; others have multiple acceptable answers; others have only criteria for what good looks like. The harness should support all three.
A pattern where another LLM scores outputs against criteria. The judge model gets the original input, the system's output, and a rubric, then produces a score. Works well for subjective quality dimensions where exact match does not apply.
Calibration matters. Check that judge scores correlate with human judgment for your use case before relying on them. Some teams use a more capable model as judge than the one being evaluated; others use the same model. The trade-off is cost versus alignment with human judgment.
On every meaningful change to the system: prompt updates, model upgrades, retrieval changes, tool definition updates. Many teams integrate the harness into CI so changes cannot deploy without running it. Production sampling adds periodic runs (daily or weekly) against current production behavior.
The cadence depends on how often the system changes and how critical the changes are. Active development cycles benefit from CI integration. Stable systems benefit from periodic production sampling.
Two metrics layers. Retrieval quality measures whether the right chunks were returned: precision, recall, mean reciprocal rank against a labeled set. Generation quality measures whether the model answered correctly given the retrieved chunks: faithfulness to context, answer relevance, completeness.
Tools like Ragas implement both layers. Most teams underinvest in retrieval evaluation and over-focus on generation, missing the upstream cause of many quality issues. Building separate evaluation for retrieval and generation surfaces this earlier.
For a 100-case harness with a frontier model judge, each run typically costs a few dollars to a few tens of dollars depending on prompt and judge sizes. Daily runs are economical for most teams. The cost is usually trivial relative to the value of catching regressions before deployment.
Cost optimization for evaluation includes using smaller models as judges where they suffice, caching evaluation results when inputs do not change, and parallelizing evaluation runs. The cost is rarely a barrier to running evaluation.
Run each test case multiple times (often 3 to 5\) and look at distributions. A model that scores well on average with low variance is more reliable than one with higher average but high variance. Set temperature to zero or low values to reduce randomness during evaluation runs.
Statistical analysis (means, variance, percentiles) gives a fairer picture than point estimates. When comparing configurations, look at distributions and use statistical tests rather than picking a winner from single runs.
When a specific failure case is found in production, add it to the harness as a regression test. Future changes that reintroduce the failure get caught before deployment. Over time the regression test set becomes a record of issues the team has solved, ensuring they do not recur.
The pattern is similar to traditional software regression testing. The accumulation of regression tests over time is one of the things that makes mature systems more reliable than newer ones; the test set encodes hard-won lessons about what can go wrong.
In addition to output quality, capture and score the agent's path: did it use the right tools, in a reasonable order, within step and cost budgets, with appropriate handoff to humans when needed. Tools like LangSmith and Braintrust support agent-specific evaluation.
The evaluation is more involved than for simple chat but follows the same pattern of representative cases plus scoring. The added dimensions (path quality, tool selection, budget adherence) require additional scoring criteria but the basic structure is the same.
In version control alongside the code. Treat them like other code artifacts: reviewed when added or changed, owned by a specific team, refactored over time. Some teams use spreadsheets early and migrate to JSON or YAML in version control as the harness matures.
The format matters less than the discipline of treating test cases as durable assets. Test cases that disappear when team members leave are less valuable than test cases stored systematically that survive personnel changes.
