Nearly everybody has a platform, so the variance that matters is entirely in quality. This instrument scores one shared ML platform and its serving tier out of a hundred points across four domains: reproducibility and lineage, the path to production, serving reliability and capacity, and tenancy and cost attribution.
Twenty rows, each naming the artefact that satisfies it, and three bands. It measures what the platform hands a team by default rather than what one determined team managed to assemble on top of it.
Four domains of twenty-five points each will average out, and the average is the least useful thing the instrument produces. Read the lowest domain first, then place the total in one of three bands and start where that band points.
Seventy-six to one hundred. Teams get reproducibility, promotion and capacity from the platform rather than from their own ingenuity, and a new team reaches production without a sponsor. The points still missing sit in the serving tier, and those are the expensive ones to buy back.
Forty-eight to seventy-five. The platform works for the teams who know it well and quietly costs everyone else a week per release. Find the two domains scoring under twelve and start with the path to production, since every team walks it on every change rather than once a quarter.
Under forty-eight. What exists is shared infrastructure with a wiki attached, and each team is privately rebuilding the missing half. Start with reproducibility and lineage, because no other domain can be verified until that one is true. Everything above it rests on a lineage query that returns an answer.
Re-run a six-month-old training job as somebody who did not author it and compare artefact hashes. Then resolve a served model picked at random to its run, dataset snapshot and code commit in one query.
Time the median gap from registry publish to first production request across the last twenty promotions. Then have a stream-aligned team run a rollback drill on a live endpoint with nobody helping.
Ask each pool for a dated headroom figure against observed peak, a measured first response on a scaled-to-zero pool, and a routing fallback that somebody has triggered in a drill.
Sample a serving trace and look for tenant, team and model version on one span. Every other row in the domain depends on that, and most platforms lose it at the gateway.
Once per platform. The unit is the shared thing every team depends on, and scoring per model or per team produces an average of individual ingenuity instead. Where a statement would hold equally for one system run by one team, it belongs to a different instrument and not to this one.
Half marks let a platform total seventy while a team still cannot promote a model on a Friday. Each row names an artefact or a measurement, and the row is satisfied or it is not. Nobody producing that artefact inside an hour is the same answer as nobody having it.
The worked platform scores 56 and is in trouble, which is the point of publishing it. Serving scores well because the platform team instrumented what it operates, the promotion path scores badly because rollback still runs through an engineer, and tenancy scores worst. Those figures illustrate a shape rather than report an observed platform.
Tenancy and cost attribution, closely followed by the path to production. Strong serving and a weak promotion path is the common shape and it has an obvious cause: the serving tier is what the platform team builds for itself, and the promotion path is where every other team has to live.
This scores a platform you already run, domain by domain, against the artefact each row needs. MLOps and Model Serving: What Buyers Should Ask covers the platform you have not bought yet, and what a seller will or will not put in writing before a contract is signed.
A VP of platform or head of ML infrastructure, sitting with one engineer from a team that does not own the platform. The second person is what keeps the scoring honest, since the rows that matter most describe what the platform gives somebody who did not build it.
Drop your details and we'll send Benchmarking MLOps and Model Serving straight to your inbox - no spam, unsubscribe anytime.
Bring last month's attribution report and the promotion path your teams actually walk. We will work two domains through with you and name the zero that is cheapest to close first. SECTION 7 - FAQ - 5 to 8 questions
Book a platform review