whitepaper

The Economics of Model Evaluation.

Evaluation arrives at a budget meeting as a quote per cycle, and the quote is accurate. It is also about a tenth of what the thing costs. What the cycle consumes is a labelled set: items somebody wrote, labels somebody applied, a rubric somebody argued over and an adjudication record for the cases that were hard.

That set is an asset with a decay rate, not a phase with an end date. This paper prices one on illustrative inputs at $78,106 in year one and $28,939 a year to keep, and finds the validation sample worth far more than the line it sits on.

Download whitepaper Talk to Our Engineers

In depth

A Cycle Got A Quote. An Asset Turned Up Instead.

The quoted figure is the smallest part of the spend.

Ask what evaluation costs and you get a per-cycle number: compute to serve the candidate across the suite, harness time to collect the results, perhaps an engineer for a morning. Underneath it sits what the cycle consumes, an eval set with items somebody wrote, labels somebody applied, a rubric somebody argued over and an adjudication record explaining the hard cases. Running the suite consumes none of that, which is exactly why it never appears beside the run cost. In the worked example the run is $8,160 against a year-one total of $78,106, a tenth of the money and the whole of the plan.

In shorta tenth of the money and the whole of the plan

Labelling was approved once and it behaves like a subscription.

A set is funded as a build, signed off, and then expected to serve for as long as anybody needs a number out of it. Two things stop that working: public and semi-public items leak into training corpora, so a score drifts upward for reasons that have nothing to do with the system improving, and the production distribution moves underneath the set as new segments and new failure shapes arrive. Neither failure announces itself, since a contaminated set and a stale set both keep returning numbers that look fine. Rotation is the only defence, and at a refresh share of a quarter it costs $13,539 a year and belongs in the operating budget with a renewal date on it.

In shortRotation is the only defence, and at a refresh share…

A judge replaced the labellers and nothing was funded to check the judge.

Automated judging is adopted for a sound reason, since human labelling at scale is the expensive line, but the saving is real only once the judge has been checked against human labels on a held-out sample, in a statistic that accounts for chance, with position randomisation applied. Skip that and the measurement becomes an assertion: the judge still returns a score, the dashboard still turns green, and nobody can say what the score is evidence of. Testing 21 judges from nine providers, UC Berkeley put a mean of 38.6 percentage points between exact-match agreement and kappa on MT-Bench, so the judge you bought at 85% is a moderate one priced as a strong one. Taking the validation sample from 200 items to 800 costs $7,440, which is 9.5% of the first year and the cheapest thing on the sheet.

In short5% of the first year and the cheapest thing on the sheet

A score without an interval turns noise into a roadmap.

Two models come back at 71.4% and 73.1% on a 400-item set, the newer one wins by 1.7 points, and a quarter of engineering effort gets pointed at whatever produced the gain. On a set that size the ninety-five per cent interval around a proportion near the middle of the range runs roughly five points either way, so the two results are indistinguishable and the quarter went on sampling variation. This is the cheapest of the four errors to correct and the most expensive to leave alone, since its cost lands in the engineering budget rather than the evaluation one.

In shortThis is the cheapest of the four errors to correct a…

The detail

Three Lines A Finance Review Trims First, Ranked By What They Cost You.

A plan that prices the run carries two of six cost lines. The four it leaves out are all tied to the set rather than to the suite, and that is where the growth lives. These three decide whether anybody can defend the number at the end of it.

Zone · 01

Judge validation

The line cut first, on the reasoning that it buys no coverage. Take the sample from 200 items to 800, a fourfold swing nobody would concede in a planning meeting, and the year-one total moves $7,440, or 9.5% of it. That spend is what tells you whether $78,106 bought a measurement or a guess, and it halves the interval around every score while doing so.

Zone · 02

Set refresh

The term that converts evaluation from a project into an operating cost. At a refresh share of a quarter the set turns over in four years, slower than most production distributions move, and it carries $13,539 a year in the worked example. Setting that share to zero is a decision about contamination, and somebody senior should be asked to sign it rather than inherit it.

Zone · 03

The disagreement rate

A property of the rubric rather than of the raters, and the one input nobody writes down. An unclear rubric raises the adjudication line twice over, once through the volume of disputed items and once through the senior minutes each case takes to settle. When it climbs, rewrite the rubric instead of paying senior time to have the same argument item by item.

By the numbers

The figures that make it a board-level conversation.

10%

share of the first-year evaluation total that is the eval run everybody quotes

9.5%

how far quadrupling the judge validation sample moves the year-one total

0.48

the chance-corrected figure underneath an 85% MT-Bench agreement rate, which is the one nobody prices

Inside the report

What you'll take away.

Step 1 - Put the eval set on the asset register with a named owner

Items, labels, rubric and adjudication record belong to one accountable person with a refresh budget, not to whichever project last needed a number out of them.

Step 2 - Buy the validation sample before you buy the judge

A judge adopted without a held-out human sample converts a measurement into an assertion. Report kappa with position randomisation applied, and re-run it whenever the judge version changes.

Step 3 - Fund rater depth before item count

Three raters on 1,200 items gives you an agreement figure you can defend. One rater on 3,600 items gives you none, and costs more to produce while giving it.

Step 4 - Write the refresh share into the operating budget and date the set

Decide what percentage rotates each year before the build project closes. A set with no rotation plan still has an expiry date, just an undeclared one nobody owns.

Questions

Frequently asked.

What does an evaluation programme actually cost in its first year?

In the illustrative worked example, $78,106 for 2,400 items at three raters each. The eval run everybody quotes is $8,160 of that. The set build is $62,706 and judge validation $7,240, and steady state afterwards runs $28,939 a year.

The shape of the arithmetic is what transfers to your programme; none of the inputs does.

Which of these numbers came from research rather than from your model?

One of them. The kappa behind an 85% agreement rate, roughly 0.48, comes from UC Berkeley's 2026 work on 21 judges, run over MT-Bench, JudgeBench and RewardBench. Every cost input here is illustrative: 2,400 items, three raters, $62 a labeller hour, $95 a senior hour and 24 release cycles a year.

Why defend the validation sample rather than the set size?

Defend both, but know what each one buys. Running the item count from 1,200 to 3,600 moves the year-one total 7.3 times as far as a fourfold swing in the validation sample does.

The validation sample is the small line that tells you whether everything above it measured anything at all.

Our judge reports 85% agreement with our own raters. Is that good?

It is the flattering way of saying moderate. In the Berkeley cohort the two statistics sit a mean of 38.6 points apart, which leaves an 85% headline standing on a kappa of about 0.48. Ask for the chance-corrected figure, with position randomisation applied, before any threshold is set on it.

We upgraded the judge model. Does the earlier validation still hold?

No, and neither do the scores that judge produced. Each earlier validation describes a system that no longer exists. Most of the 21 judges Berkeley tested changed rank by at least four places when the benchmark changed, so treat a swap as a re-baselining event and budget validation per judge version.

Does this replace scoring the evaluation system itself?

No, it sits beside it. This paper puts a price on the apparatus: what a labelled set costs to build, to validate and to keep current. Benchmarking Model Evaluation grades the same apparatus and locates its weakest row.

One of them tells you what to fund, the other what to fix.

Whose numbers does this model depend on?

Two sets of them, held on opposite sides of the building. One holds the item count, the rubric and an honest view of how often two careful raters disagree. The other holds the loaded rates, the renewal cycle and the question of which budget a refresh is paid out of.