A vendor quotes agreement because agreement is the kind statistic. It credits the judge for every verdict chance alone would have produced, and across 21 judge models from nine providers the two figures differ by 38.6 percentage points on average.
A judge sold to you at 85% agreement is sitting near a kappa of 0.48. This playbook sets twenty-one questions across nine areas of the contract, each one printed against the answer that ends the conversation and the answer that earns the next meeting.
Written replies stay claims until a product meets data it has never seen. Two weeks of measured comparison is worth more than a quarter of written assurance, and it is worth nothing unless you own the ground truth. Three terms decide whether the trial produces evidence.
Label three hundred of your own items before any vendor sees a row: two labellers on a stated overlap, disagreements adjudicated by a named person, and your own inter-rater kappa computed and written down. That figure is the ceiling on what any judge can be validated to, and a bidder scoring above it has scored noise.
Split your set by difficulty, release the easy portion for tuning, and score both bidders on the hard remainder nobody has touched. A product tuned on your examples will tell you how well it tunes, which is not what you are buying. Keep the sealed half sealed until both submissions are in.
Present every pairwise case in both orders and include matched long and short versions of the same answer. Record the distance of P(A wins) from an even split and the length-to-verdict correlation as scored outputs rather than footnotes. The Berkeley cohort kept length effects under 0.011, so a high figure here is a product defect and not an industry condition.
State the statistic in the request so no bidder gets to choose it afterwards. Read the gap between the two numbers as a direct measure of how much chance was carrying the headline.
A named human panel on a held-out sample, with the item count, the rubric they worked from and the agreement among those humans stated first. Validation against another model's verdicts is not validation.
A judge version you can name and hold for a contractual window, advance notice before the model behind it changes, and a bridging run on a fixed sample so the offset is published rather than discovered.
Run the export during the trial, not during the dispute. Per-item provenance, labeller identity, rubric version and the adjudication record should travel with the data as fields, in a format another supplier can run.
Agreement credits the judge for every verdict chance alone would have produced. Across 21 judges from nine providers the two statistics differed by 33.8 to 41.3 percentage points on MT-Bench, a mean of 38.6. Accept agreement and you have bought roughly forty points of comfort that nobody measured.
No, ask what it does when the benchmark underneath it changes. Kappa across the Berkeley cohort spread 13.5 points on MT-Bench against 60.4 on JudgeBench, about 4.5 times wider, and more than half of the 21 judges moved four or more ranking positions between benchmarks. One figure describes one test.
It settles reproducibility and nothing else. Berkeley found the two properties coming apart in the same cohort: judges whose test-retest reliability cleared 0.95 while position effects on those same judges reached 0.192. A steady judge makes the same error every week, which is why nobody catches it.
Unless somebody drew a boundary on the chart, it is worth less than it looks. Ask for the judge treated as a release unit: advance notice, both versions addressable at once, a bridging run on a fixed sample, and the measured offset published rather than left for you to find.
Every question here is one we expect to be asked ourselves, and two of them we answer less comfortably than the rest. A guide that skipped those would not be worth reading.
Send the whole set to us on the same terms as to every other bidder, and compare what comes back.
Only at the edges. This one reads what a vendor commits to on paper while you can still walk away. Benchmarking Model Evaluation is the instrument for the apparatus you already operate, scoring the set it uses, what its judge is actually measuring, and whether any of that ever predicted production.
Read that one once the contract is signed and the apparatus is yours.
A director of QA, with whoever will run the suite daily once it is bought. Add the person who signs the data agreement before area seven, since ownership of the labelled set and its adjudication history is a legal question as much as a technical one.
Drop your details and we'll send Model Evaluation: What Buyers Should Ask straight to your inbox - no spam, unsubscribe anytime.
Bring the judge a vendor is currently asking you to buy and a few hundred of your own adjudicated items. We will restate its headline figure in kappa with an interval, in front of you, before anybody talks price. SECTION 7 - FAQ - 5 to 8 questions
Review an evaluation vendor