framework

5 min read

MLOps and Model Serving Vendor Scorecard.

Choosing an MLOps or model serving vendor is a multi-year commitment that looks like a tooling choice. Pick on demos and list prices and you tend to pay twice: once in a cost per prediction that never matches the slide, and again in the migration when you outgrow the platform.

This scorecard makes the comparison fair. It weights eight categories by business impact, defines what a 1, 3 and 5 look like, and pairs the scores with a ten-day proof of concept run on your own model and traffic.

It rates no named vendor, and every score comes from your own evaluation.

Download framework Talk to Our Engineers

In depth

Why Platforms Look Better in the Demo Than on the Invoice.

A vendor's benchmark measures the vendor's chosen workload.

Throughput figures are real, but they come from the model, precision, batch size and prompt length the vendor picked. Your traffic differs, so the scorecard scores performance only on throughput measured in your proof of concept at your latency SLO, and a datasheet claim scores a 1 until it is demonstrated. Evidence beats adjectives.

In shortEvidence beats adjectives

Price per hour is not price per prediction.

A platform with a higher hourly rate can be cheaper if it serves more tokens per GPU or scales down faster, and per-token pricing can be dearer at steady volume. The scorecard asks for an all-in cost per 1M tokens or per 1K predictions, calculated from proof-of-concept usage and quoted in writing at your forecast volume. If the bill cannot be calculated before signing, it will surprise you after.

In shortit will surprise you after

Capacity promises are only promises until they are in the contract.

GPU availability in your regions, reserved capacity and failover are where serving platforms differ most under pressure, and where verbal assurances count for least. Reliability and scale carry 15% of the weight, scored on contractual SLAs and documented, testable failover. Write the capacity commitment down or assume it does not exist.

In shortWrite the capacity commitment down or assume it does…

Exit costs are cheapest to measure on the way in.

Proprietary model formats, closed APIs and platform-specific pipelines all raise the price of leaving, so the scorecard asks every vendor how many days it would take to move your models elsewhere and scores open formats higher. Portability carries only 5% of the weight, but a 1 on any mandatory criterion removes a vendor whatever its total. Leverage is highest before signature.

In shortLeverage is highest before signature

The detail

Three Rules That Keep the Evaluation in Your Hands.

Vendors are good at steering evaluations toward the criteria they win on. Three rules keep control with the buyer, and each produces a document that can be handed to procurement.

Zone · 01

Set weights before vendors arrive

Agree the eight category weights with engineering, finance and security before the first call. The defaults put performance and cost at 20% each and reliability and time to production at 15%, but your priorities may differ. The artefact is a dated weighting table signed off by stakeholders.

Zone · 02

Score only what was shown

A 5 means proven in your proof of concept or committed in the contract, a 3 means documented but unproven on your workload, and a 1 means missing, roadmap-only or claimed without evidence. The artefact is a scorecard where every score above 1 has a linked piece of evidence.

Zone · 03

Test every vendor on your traffic

Deploy your model, replay real requests at expected and twice peak load, spike to five times, run a canary and a rollback, export metrics and calculate the all-in unit cost, with the same tests for every vendor. The artefact is a side-by-side results table, one column per vendor.

By the numbers

The figures that make it a board-level conversation.

weighted categories, each scored only on evidence from your own proof of concept

10 days

of structured testing on your model and traffic before any contract is signed

$2.82

per H100 GPU-hour on the neocloud rental index, a reference point for negotiation

Inside the report

What you'll take away.

Step 1 - Agree the category weights with your stakeholders

Settle the eight weights with engineering, finance and security before contacting any vendor, and record which criteria are mandatory.

Step 2 - Send every shortlisted vendor the same five questions

All-in monthly cost at forecast volume, guaranteed GPU capacity, data use and retention, exit effort and a reference customer at your scale.

Step 3 - Run the ten-day proof of concept on your own workload

Same model, same tests and the same traffic replay for every vendor, with throughput, scaling, rollback and unit cost recorded.

Step 4 - Score on evidence and take the gaps into negotiation

Calculate weighted totals, remove any vendor with a 1 on a mandatory criterion and use measured differences in commercial talks.

Questions

Frequently asked.

Which vendors does the scorecard cover?

Any platform that trains, deploys or serves models: hyperscaler ML platforms, data and AI platforms, specialist inference providers and open-source stacks with enterprise support. It deliberately scores no named vendor, since the result depends on your workload.

Why do performance and cost carry the most weight?

Because they decide the monthly bill for as long as the platform runs. AWS has put inference at up to 90% of production ML compute, so a small difference in throughput per GPU or unit price compounds every month.

Adjust the weights if data residency or speed to market matter more to you.

Can we skip the proof of concept if a vendor has strong references?

References tell you a platform works for someone else. The proof of concept tells you what it costs and how it behaves on your model and traffic, and ten days of testing is small next to a multi-year commitment.

How should we treat open-source options?

Score them with the same scorecard, treating the support provider as the vendor. Open-source stacks usually score well on portability and on unit cost at scale, and the honest test is whether your team or a support partner can operate them reliably.

What are the red flags?

Benchmarks only on the vendor's chosen model and settings, pricing that cannot be calculated from your traffic before signing, proprietary formats with no export path and capacity promises that are missing from the contract.

What if two vendors finish close together?

Look at the mandatory criteria and the exit cost first, then at the gap in all-in unit cost at your forecast volume. Measured differences from the proof of concept are the strongest negotiating points you will have with either vendor.

How long does the whole evaluation take?

About three to four weeks: a week to agree weights and send questions, ten working days of proof of concept and a scoring session. That is short next to the cost of migrating off the wrong platform later.