An automated judge quoted at 85% agreement with human raters is sitting at a Cohen's kappa near 0.48, and almost nobody reports the second number. A judge is a model running in production, with a version, a prompt and a decoding setting, and it decides what every other model is allowed to claim.
Twenty-seven numbered controls across seven families: the evaluation set, the rubric, human labelling, judge validation, statistical discipline, contamination and the production feedback loop. Every row names an object somebody has to build and the person whose backlog it lands in.
The families run in build order and each one caps the next. A rubric cannot be more precise than the task under it, human agreement cannot exceed what the rubric makes decidable, and no judge can be validated past the quality of the labels it is checked against.
human labelling
One labeller per item means no agreement statistic exists, and the ceiling on what any judge can later be validated against is unknowable. A stated fraction of each batch is multiply labelled, agreement is reported in Cohen's kappa or Krippendorff's alpha with an interval, and a batch below the written floor stops the run and reopens the rubric.
judge validation
Six controls, and the one that carries them is EVL-4.6: a human sample continues after the judge is adopted, sized to detect drift in the judge rather than in the system. The judge is never the model under test, and same-provider use is declared. Pairwise comparisons randomise order and publish position bias each release.
the feedback loop
An offline score nobody checked against an outcome is a reassurance. Correlate it against something that matters, such as task completion, escalation, human override or reversal, and refresh that correlation quarterly. Every class in the failure taxonomy carries an item count, and a class seen in production but absent from the set is a coverage defect with an owner.
A held-out human-labelled sample drawn from the same population the judge will go on to score, with the sample size and the draw record kept. Kappa with an interval, agreement beside it.
A new checkpoint, a prompt change or a different decoding setting re-opens validation and gets its own release record: version, prompt hash, decoding settings and validation date. Scores either side are not comparable.
Compute the minimum detectable difference before sizing, carry a confidence interval on every reported score, and count repeated looks at a held-out set against a stated budget. A set queried on every commit is not held out.
Canary strings, perturbed variants or a membership check, run on a fixed cadence and again on every model change. Give each set a refresh cadence and a retirement date the day it is built.
EVL-1 and EVL-2, in that order. A sampling frame decides which population your number describes, and a written rubric decides whether human labels mean anything. Judge validation is wasted effort until both exist, since no judge can be validated past the quality of the labels it is checked against.
Agreement counts the cases two raters would have matched by chance. Berkeley's cohort of 21 judges over roughly 541,000 judgments put exact-match agreement 33.8 to 41.3 points above Cohen's kappa on MT-Bench, mean 38.6, so 85% agreement corresponds to a kappa near 0.48. Report both and gate on kappa.
No. The same study found test-retest reliability above 0.95 coexisting with severe position bias, measured as the distance of P(A wins) from 0.5 and running from 0.002 to 0.192 across the cohort.
A judge that returns the same wrong verdict every time is the hardest kind to catch, and its consistency is what people cite in its defence.
Not reliably. Kappa spanned 13.5 points across the cohort on MT-Bench and 60.4 on JudgeBench, roughly 4.5 times wider, and more than half the 21 judges moved four or more ranking positions between the two. Validate on a held-out sample from the population your judge will actually score.
Check contamination and check the loop. A held-out set that has reached a training corpus produces a rising figure and no improvement, which is the most expensive kind of wrong number. Test by canary string, perturbed variant or membership check on a fixed cadence, and correlate the offline score against a production outcome such as escalation or human override.
Only where a control produces the evidence. Model Evaluation Under Regulation sets out what Article 15 obliges a provider to declare, what Annex IV wants said about the method, and why no harmonised standard yet settles either.
Read it first when a conformity assessment is the reason the work exists; everything here is the build underneath it.
Evaluation leads, QA and ML engineers, and whoever owns the release gate. It assumes nobody has asked you for this apparatus yet and that you are building it anyway, and that your arguments are about sampling frames, agreement floors and which judge version a figure was computed under.
Drop your details and we'll send Model Evaluation: An Engineering Reference straight to your inbox - no spam, unsubscribe anytime.
Point a two-week trial sprint at EVL-4. Bring a few hundred of your own labelled items and we hand back the kappa for whichever judge you already report from, next to the agreement figure you have been quoting. SECTION 7 - FAQ - 5 to 8 questions
Book an evaluation design review