Article 15 obliges a provider to state the accuracy levels of a high-risk system and the metrics behind them. Annex IV obliges the technical file to describe how those levels were established.
Neither says which metric, measured on which population, against which threshold, or judged by whom. The harmonised standard written to settle that, prEN 18229-2, was at draft stage at CEN-CENELEC JTC 21 as of the most recent public information.
So a provider shipping in 2027 makes three choices nobody has ratified, and defends them later against a document that did not exist when the choices were made.
Seven obligations in the paper are restated as demands on the measurement apparatus rather than on the system as a whole. These three are what a notified body or an enterprise auditor asks for first, and none of them can be produced after the fact.
Article 15 wants a level and its relevant metrics in the instructions for use, which means a formula, the scoring rules for ties and refusals, the eval set version, the sample size and the interval around the figure. Where a judge produced the number, state kappa and say so. Dated and owned, or it is a percentage.
Article 10 asks that testing sets be relevant, representative and examined for bias against the setting the system will be used in. That is answerable only item by item: where it came from, who approved it, which deployment segment it stands for, and whether anyone checked it against the training corpus. Record it at admission.
Substantial modification turns on being able to say which change moved which number and by how much. Pin the eval set version, the rubric version, the judge version and its validation to every figure you declare, with a date. Releases then produce deltas you can show, instead of an assertion that nothing material happened.
Formula, scoring rules, eval set version, sample size and interval travel with the figure into the instructions for use. Where a judge produced it, report kappa and name the statistic you used.
Source, approver, deployment segment and a contamination check against the training corpus, recorded per item. This is the single artefact in the file that no later effort can recreate at any price.
Name what a failure costs the person at the other end, which failure direction is worse, what the fallback does when the gate fails, and who signed. That part is assessable today.
Why this metric for this harm, why this population, why this threshold, why this judge and what validated it. A recorded rationale maps onto a standard the week one is cited.
No. Article 15 binds whether or not a method has been ratified, and the declaration is due at conformity assessment. prEN 18229-2 and the related JTC 21 drafts were at draft stage as of the most recent public information, and a standard confers a presumption of conformity only once the Commission cites it in the Official Journal.
prEN 18229-2 for accuracy, prEN ISO/IEC 23282 for natural language processing evaluation methods, prEN 18281 for computer vision and prEN 18286 for the AI quality management system. All were at draft stage as of the most recent public information, and status moves, so check it rather than quoting this page in a year.
It is the flattering statistic. Berkeley's cohort of 21 judge models across roughly 541,000 judgments put exact-match agreement 33.8 to 41.3 points above Cohen's kappa on MT-Bench, so 85% agreement corresponds to a kappa near 0.48. Declare kappa, show agreement beside it, and name which one the threshold applies to.
It becomes evidence rather than fixture code. Article 10 asks whether the test population was representative of the deployment population, and only per item provenance answers that: source, approver, the segment the item stands for, and a contamination check against the training corpus.
Start recording it at admission, since the existing file cannot be back-filled honestly.
Whenever something that moved it changes: a prompt rewrite, a refreshed retrieval corpus, a new base checkpoint from the provider, a new locale, a shift in traffic mix. Version the eval set, the rubric and the judge, pin all three to every figure you declare, and the substantial modification question becomes a query instead of an argument.
This report is about what you must declare and later defend, and what a regulator or a customer's auditor can ask you for. Model Evaluation: An Engineering Reference numbers the control families that produce those figures, from the sampling frame through judge validation to the production feedback loop.
Take this one when a declaration is due, that one when you are building the harness.
Directors of QA and heads of risk who have to sign off a figure that goes into the instructions for use. It assumes you already run evaluations and that the open question is which choices you made on your own authority, and what you kept that proves you made them deliberately.
Drop your details and we'll send Model Evaluation Under Regulation straight to your inbox - no spam, unsubscribe anytime.
Point a two-week trial sprint at the figure you are about to declare, and find out what your evaluation set can prove about the population it was supposed to represent. SECTION 7 - FAQ - 5 to 8 questions
Book an evaluation evidence review