Eval as Ceremony Versus Eval as Engineering
Most enterprise LLM evaluation is ceremony. The team runs the model against a small set of test cases before launch. The results look good. The launch proceeds. Once in production, the model's behavior changes and the team finds out from customer reports.
The alternative is eval as engineering: a harness that runs continuously, blocks releases that regress, and produces measurable evidence of quality over time. The harness is not glamorous and is the difference between sustainable AI engineering and reactive AI operations.
OpenAI's evaluations framework, Anthropic's published eval methodology, and the open-source DeepEval project all document patterns that have settled in 2024-2025 into a recognizable internal eval harness architecture (Anthropic, "Building evaluations," 2024; OpenAI Evals on GitHub, 2024). The pattern has five components.
The Five Components
A production-grade internal eval harness has five components. Each one is engineering investment. Each one fails predictably when omitted.
The first component is the eval set. A curated collection of inputs with expected behaviors. Not random samples. Stratified samples covering high-frequency patterns, known edge cases, customer-flagged failures, and the specific behaviors the system needs to maintain. The eval set is the contract.
The second component is the rubric. For each input, what counts as a correct response. Sometimes exact match. More often a structured rubric describing the dimensions of correctness (accuracy, format, tone, refusal handling, citation behavior). The rubric is what makes evaluation reproducible across runs and reviewers.
The third component is the grader. The mechanism that applies the rubric to model outputs and produces scores. Three grader types are common: programmatic for exact-match or schema-validated outputs, LLM-as-judge for evaluations that require language understanding, and human review for the highest-stakes outputs. Most production harnesses use a combination.
The fourth component is the harness orchestration. The infrastructure that runs the eval set against models, collects outputs, applies the graders, aggregates scores, and produces reports. Modern harnesses integrate with CI/CD so evaluation runs on every relevant change.
The fifth component is the alerting and dashboarding layer. The interface that makes eval results visible to engineering, product, and operations stakeholders. Without visibility, the eval results exist and do not influence decisions.
A harness with all five components running is engineering-grade. A harness with three or four is partial and produces partial benefits.
What Each Component Costs to Build
The components have different build complexity and recurring cost.
The eval set is the most expensive to build well and the cheapest to operate. Building requires deliberate curation, often over several weeks, involving subject matter experts and engineering judgment. Operating requires version control and periodic updates. The eval set is also the component that most teams underinvest in, building it once and then neglecting it.
The rubric is similar. Designing it well takes time and judgment. Operating it requires updates as the system evolves.
The grader is moderate complexity. Programmatic graders are simple engineering. LLM-as-judge graders require their own meta-evaluation: how reliable is the judge model's grading. Human review graders require process design for who reviews what at what cadence.
The harness orchestration is the most code-heavy component but uses mature tooling. Open-source frameworks (DeepEval, RAGAS, the LangSmith eval harness) and commercial platforms (Braintrust, Galileo, Arize, Langfuse) all provide harness orchestration. Most teams in 2026 use one of these rather than building from scratch.
The alerting and dashboarding is straightforward engineering using standard observability tools (Grafana, Datadog, custom dashboards). The work is presenting eval results in the same operational interfaces the team already uses.
Total buildout for a production-grade harness is typically 6-10 engineering weeks for initial setup plus 10-15 percent of one engineer's ongoing capacity for sustained operation.
The Eval Set Construction Discipline
Of the five components, the eval set is where most quality variation comes from. A bad eval set with a good harness produces uninformative results. A good eval set with a minimal harness produces useful results.
Three properties distinguish good eval sets.
Representativeness. The eval set reflects what the model actually encounters in production. For pre-launch systems, this requires inference from user research and pilot data. For production systems, this requires sampling from real production traffic.
Coverage. The eval set covers the dimensions of behavior that matter. High-frequency intents, known edge cases, the specific behaviors that customers complain about when they break. Coverage is broader than accuracy: it includes tone, format, refusal behavior, latency requirements.
Maintainability. The eval set has a documented process for additions, deprecations, and modifications. Examples have version history. New examples can be added through normal engineering review. The eval set evolves with the system rather than freezing at launch.
Eval sets that hit these three properties produce useful measurement. Eval sets that miss any of them produce noise.
The LLM-as-Judge Question
LLM-as-judge grading has become standard in 2024-2025 because it scales human-level evaluation across more inputs than human review can cover. The practice raises legitimate questions.
The first question is reliability. How consistent is the judge model's grading. The answer depends on rubric design and judge model selection. Well-designed rubrics with strong judge models (typically a frontier model judging outputs from a smaller production model) produce consistency in the 85-95 percent range against human gold-standard grading.
The second question is bias. Judge models have their own biases that affect grading. The biases are documented (preference for verbose responses, particular formatting preferences, language patterns). Awareness of the biases shapes rubric design to compensate.
The third question is calibration. LLM-as-judge grades drift over time as judge models update and prompt patterns evolve. Periodic calibration against human grading maintains reliability.
LLM-as-judge is useful when implemented with these caveats. The practice fails when teams treat the judge as ground truth without verification.
What the Harness Catches Versus What Slips Through
A well-functioning harness catches quality regressions, schema violations, refusal pattern shifts, and most common failure modes. The harness does not catch everything.
Issues that are present in production but absent from the eval set slip through. The eval set's coverage is the gating constraint.
Issues that manifest at scale but not in eval runs slip through. Eval typically runs on a sample of inputs; production runs on many more inputs.
Issues that depend on conversational context (when the model behaves correctly individually and incorrectly in conversation) slip through unless the eval set includes multi-turn evaluation.
Issues that require human judgment to recognize (some refusal patterns, some tone issues) slip through automated grading.
The harness reduces the surface area of problems that escape to production. It does not eliminate it. Combining the harness with production monitoring (sampling eval running against live traffic) catches more than either alone.
What Logiciel Does Here
Logiciel works with engineering teams building or upgrading their internal eval harness for production AI workloads. The work is typically structured around the five-component buildout with priority on eval set construction because the eval set quality determines harness value.
The AI Testing and QA Automation framework covers the inverted-pyramid testing approach that the eval harness sits within. The Production-Grade AI Implementation framework covers the six characteristics that the harness enables.
A 30-minute working session is enough to assess your current eval discipline against the five-component reference.
Frequently Asked Questions
How big should my eval set be?
100-500 examples for most production workloads. Smaller is brittle. Larger costs more to run in CI and tends to get neglected. The quality of examples matters more than the count.
Should I build the harness or buy?
For the orchestration component, buy. The frameworks have matured to the point that internal builds rarely outperform. For the eval set and rubric, build. Your specific workload needs your specific evaluation.
How often should the eval run?
On every prompt change. On every model update. On every retrieval configuration change. On a daily schedule against production. The cadence reflects how often the system changes.
What is the right team to own the eval harness?
Engineering owns the harness infrastructure. Subject matter experts own the eval set content. Product owns the rubric definition. Pure engineering ownership produces evals disconnected from product needs. Pure product ownership produces evals that engineering cannot maintain.
How do I demonstrate ROI for the eval harness?
Through prevented incidents. Each quality regression caught in CI is an incident that did not reach customers. Track these explicitly. The ROI conversation becomes straightforward when the prevented-incident count is in front of the CFO. Sources: - Anthropic, "Building evaluations," 2024 - OpenAI Evals on GitHub, 2024
