Almost every quality claim an organisation makes about its AI rests on a measurement taken by a second model, and that measurement is published in the wrong units. A study of 21 judge models across three public benchmarks finds raw agreement with human raters sitting around 38 points clear of Cohen's kappa, which leaves a judge published at 85% agreement carrying a kappa around 0.48.
The same work finds near-perfect consistency coexisting with severe position bias. That is one study of judges on benchmarks, not a survey of how enterprises run evaluation, and the difference matters.
None of this asks a board to become statistically literate. It asks a board to notice which of two reporting habits it is being shown, and that is legible on the page. Three questions do it, each answerable with one named artefact.
Ask what the agreement figure actually is, and accept only an answer that names a statistic. A room that cannot say whether its headline number is raw agreement or a chance-corrected one is reporting the flattering version, since that is the default and takes no decision to produce. The artefact is the reporting template with both columns on it.
Ask which held-out sample of human labels the judge was checked against, who made it and when anybody last touched it. A judge validated against nothing is not a measurement instrument, it is a second opinion from a system carrying the same blind spots as the first. The artefact is one page per judge in production, or a written statement that none exists.
Ask what becomes of every historical score when the underlying judge is moved to a new checkpoint. A series nobody re-baselined is comparing two instruments and calling the difference progress. The honest answer is that the series breaks and has to restart, and very few organisations have ever said so aloud. The artefact is a pinned judge version.
Raw agreement may stay for context. The reported number becomes Cohen's kappa or an equivalent, both columns appear on the template, and thresholds are set on the second one.
Kappa spread across the same judges differs by around 4.5 times between MT-Bench and JudgeBench, so the test partly decides the verdict. Write the selection rationale down beforehand.
A trend running back further than the measurement method has is not evidence of improvement. Mark every point at which the judge, the benchmark or the rubric moved, and restart the series there.
The European standards are still drafts and Annex III duties apply from 2 December 2027. Write the method, its measurement conditions and the reasoning behind its threshold, dated this quarter.
Not on this evidence. The study measures a reporting convention rather than your deployment, across 21 judge models on three public benchmarks. It says the figure most organisations publish flatters a judge by around 38 points.
Whether yours is fit for your workload is a separate question, answered on your own labels.
As far as judge models on MT-Bench, JudgeBench and RewardBench, and no further. Norman, Rivera and Hughes put 21 judges from nine providers through 118 runs, roughly 541,000 individual judgments in all. It is not a survey of enterprise practice and it locates no particular company anywhere in a distribution.
The chance-corrected equivalent, with position randomisation applied and the sample size stated. On MT-Bench an 85% agreement rate leaves a kappa of roughly 0.48, which describes a moderate judge rather than a passing one. The 85% is arithmetically correct and answers a question nobody meant to ask.
Less than it sounds. The study records consistency over 0.95 on judges that also showed a strong pull towards whichever answer came first, as weak as 0.002 in one judge and as strong as 0.192 in another.
Consistency is what stops anybody checking, so ask for the bias measurement beside it. Verbosity bias came in under 0.011.
It selected on a property that does not hold across tests. Over half the cohort shifted by four rank places or more when the benchmark changed, and your own workload resembles none of the three.
The only ranking that transfers to your task is one produced on your task, against labels your own people made.
With an instrument built for that job. Benchmarking Model Evaluation treats everything that stands between a model and a release decision as one scored system, and reports which part of it is carrying the weakness. This report explains why the statistic matters; that one says where yours stands.
Five things, each decided in a meeting and checkable afterwards by one artefact: require a chance-corrected statistic, ask what validated each judge in production, name the benchmark behind any comparison, fund the labelled set with a refresh cycle, and write down how accuracy will be declared under Article 15.
Drop your details and we'll send The Model Evaluation Report straight to your inbox - no spam, unsubscribe anytime.
Bring the AI quality figure your board pack already carries and the judge that produced it. We will work through what the same evaluation looks like reported as a chance-corrected number with a sample behind it. SECTION 7 - FAQ - 5 to 8 questions
Book a measurement review