whitepaper

Benchmarking Model Evaluation.

Precision is not validity, and an evaluation system can carry a great deal of the first while nobody has tested the second. This instrument scores one evaluation system: the held-out set, the rubric, the labellers, the judge, the statistics and the thresholds, taken together as the apparatus that decides whether a model ships.

Four domains, twenty rows, three bands. Measurement validity carries thirty-two points, the largest weight here, since an apparatus that measures the wrong property precisely is the one nobody thinks to argue with.

Download whitepaper Talk to Our Engineers

In depth

Precision Without Validity Is The Dangerous Combination, And The Common One.

Score the apparatus, not the model it passed.

One evaluation system means the held-out set, the rubric, the labellers, the judge, the statistics and the thresholds, working as one thing to answer whether something is good enough to ship. The apparatus gating your support assistant may be three people arguing over a spreadsheet while the one gating your document classifier is a pipeline nobody has opened in a year, so those are two separate exercises. The question asked twenty times over is the same one: when this apparatus returns a number, what is that number worth.

In shortwhat is that number worth

Domain C asks whether the number is stable.

Domain B asks whether it is about anything, and the two answers come apart more often than they travel together. Berkeley separated exactly these two properties: judges that reproduced their own verdicts almost perfectly were found carrying severe position bias, with order alone pushing P(A wins) as far as 0.192 from an even split across the cohort. High in C and low in B is the combination to fear: it returns the same wrong answer every Tuesday, with an interval around it, and nobody questions it because it never moves.

In shortnobody questions it because it never moves

Fill the input rows before you score a single domain.

An eval set built by the team that ships the model, from traffic captured once in 2024, reads nothing like one commissioned from outside and refreshed each quarter. The row asking when the set was last refreshed settles half of the provenance domain on its own, and the row asking who may refuse a change to the set usually settles the rest. A set that only grows measures an older product every quarter.

In shortA set that only grows measures an older product ever…

The detail

Reads The System, Reads The Easy Half, Or Reads Back What You Assumed.

Four domains of unequal weight will not average into anything you can act on. Read the band first, since each one names what the apparatus is actually reading when it returns a number, then go to the row carrying the most unawarded points inside the weakest domain.

Zone · 01

Reads the system

Seventy-five to one hundred. The apparatus measures a property that predicts production behaviour and reports it in a statistic that survives a question. What is left sits in the link to production outcomes, where an evaluation system this good has usually still never had a release blocked and therefore does not know what its gate does under pressure.

Zone · 02

Reads the easy half

Forty-six to seventy-four, where the worked evaluation system lands at 51. The arithmetic is sound and the data is clean, and nobody has checked whether the thing being counted is the thing that matters. Restate the headline figure in kappa first. It costs an afternoon and tells you how much of your confidence was chance agreement.

Zone · 03

Reads back your assumptions

Under forty-six. The apparatus returns a number that largely reflects how it was built. Put the instrument down and work on the evaluation set instead: until every item records where it came from and who labelled it, a validity result or a discipline result is only a result about whatever sample you happened to collect.

By the numbers

The figures that make it a board-level conversation.

51 / 100

the worked evaluation system, whose own team would call it mature

32 points

carried by measurement validity, the largest domain and the one most checklists leave out

0.48

Cohen's kappa under a judge whose headline reads 85% agreement, and the ten-point row that catches it

Inside the report

What you'll take away.

Step 1 - Sample thirty items and read the provenance field on each

Domain A carries twenty-eight points. Each item should name a source, a date and a labeller as fields, not a README line covering the batch. Then check the set's distribution against a recent production sample.

Step 2 - Restate the headline figure as kappa, with agreement beside it

Domain B carries thirty-two points and ten of them sit on this row alone. The next eight need the validation run itself: sample size, the human labels it was scored against, and the date it was taken.

Step 3 - Ask for the last release report and look for intervals

Domain C carries twenty-two points. A point estimate invites a decision the sample cannot support, so every headline figure needs an interval, and the minimum detectable difference has to be dated earlier than the comparison it governs.

Step 4 - Name one release this apparatus has stopped

Domain D carries eighteen points. A gate that has never held is a report with a threshold printed on it, so ask for one blocked release in the last two quarters, the figure that blocked it, and what happened next.

Questions

Frequently asked.

Why does measurement validity carry thirty-two points?

An apparatus that measures the wrong property with great precision is worse than one that is visibly noisy, since nobody argues with it. Most eval maturity checklists spend their weight on coverage and automation, both of which an apparatus can pass while measuring something that does not predict production behaviour at all.

Our apparatus gates three different systems. How do we score it?

As three exercises, one per system. Each row assumes a named system whose release this apparatus decides, a named owner of the evaluation set who can refuse a change to it, and a threshold currently in force.

Score each on its own sheet, since a shared harness rarely means a shared standard of evidence.

What does the worked example's fifty-one actually describe?

A support assistant scored against a 1,400-item set built from 2025 traffic, 310 items of it carrying a human label, last refreshed eleven months ago. Statistical discipline reaches 15 of 22 on sound arithmetic.

Measurement validity reaches 8 of 32, since the judge has never been scored against human labels. Nothing there was observed at a client.

Is partial credit really not allowed anywhere?

No row takes half marks. The right-hand column names a measurement already taken or an artefact already written, so the only live question is whether somebody can put it in front of you this week.

If it would have to be produced first, the row is a zero and that is the result.

We score well on coverage. Why is that not a domain here?

Coverage tells you how much of the surface the set touches, not whether the set resembles deployment traffic or whether the judge reading it is right. Domain A asks for a written comparison of the set's distribution against a recent production sample, which is the coverage question that actually bites.

The scoring found our holes. Where do we go next?

Start with the heaviest unawarded row rather than the cheapest. Model Evaluation: An Engineering Reference sets out the evaluation control families, each one carrying the artefact it has to produce, including those treating an automated judge as a model in production.

Score here to find out which control is missing, then read that one to build it.

Which two people make this scoring honest?

The named owner of the evaluation set, and one engineer who ships the model it gates. Without the second, the input rows get answered from memory. Whether the set's owner reports into whoever ships the model is settled by reading the chart upward, not by asking anybody how independent they feel.