A quoted p50 is one point on a curve, and the seller chose which point. Continuous batching makes latency and throughput a single dial, so a figure published without its concurrency, batch size, token shape and accelerator model cannot be placed on that curve at all.
You are buying capacity for teams who are not in the room. This playbook sets sixteen questions across nine areas beside the weak answer and the strong one, so the operating point, the idle accelerator and the checkpoint calendar are settled on paper rather than discovered in production.
Nine areas carry the sixteen questions and three of them decide the rest. Send these before the deep dive and keep what comes back, then ask for the artefact behind any strong answer rather than the sentence that delivered it.
Ask for a curve rather than a figure: p50, p95 and p99 across stated concurrency levels and batch sizes, on a named accelerator and a pinned inference server. A seller who separates queue wait from compute has run a loaded test. A seller who offers to divide the figure by replica count has run a spreadsheet.
Either dedicated and yours to fill, or pooled with a named isolation guarantee, a per-tenant floor and stated quota behaviour when a neighbour surges. A provider pooling spare capacity across tenants is making a reasonable offer and should describe what makes it safe. One charging for peak-shaped provisioning while pooling it anyway is selling the same hour twice.
Ask whether you can pin an immutable checkpoint identifier, how long you may stay on it, and what arrives in your inbox before it is withdrawn. A workable answer contains a notice window counted in months, both versions addressable at once so migration is a routing change, and an evaluation delta they publish rather than you discover.
Take the worst hour your gateway logs show that month and replay it with bursts and gaps intact. A constant request rate measures a constant request rate and nothing you will ever serve.
State concurrency, the prompt and output length distribution and the p99 target in the scope note. Then make each vendor publish every parameter they changed, including the ones that made the result worse.
Idle it for an hour, send one request and record image pull, weights load and cache warm separately. Repeat at three in the morning, when the scale-to-zero policy is actually exercised.
Take out weights, tokeniser, configuration and registry records while the relationship is cordial, then load them on a different runtime. An export nobody has executed is a clause, not a capability.
It was measured at an operating point the seller picked, and the operating point never reaches the quote. Without the concurrency, the batch size, the prompt and output length distribution and the accelerator part number, the figure cannot be placed on the curve, so it is true somewhere and useless everywhere else.
Compare structure, not rate. Inference cost fell roughly 280 times in two years, so the number you negotiate this quarter dates quickly while the contract shape outlives it. Ask what stops being billed when traffic drops to a tenth, and what is admitted when it multiplies for ninety seconds.
An immutable version identifier you may serve for a contractual window, a deprecation calendar published a release ahead of withdrawal, both versions addressable at once so migration is a routing change, and an evaluation delta they publish. An alias resolving to the current best version is not pinning.
It settles in two weeks what procurement argues about for a quarter, but only where the operating point is fixed in writing before either vendor touches a dial. Replay a real peak hour with its arrival pattern intact, and run the export yourself rather than reading the clause that describes it.
This playbook is for a platform you have not bought yet, and it works through what a seller will and will not put in writing. Benchmarking MLOps and Model Serving is the instrument for scoring a platform you already run, domain by domain, against the artefact each row needs.
VPs of platform and heads of ML infrastructure buying serving capacity that other teams' features will sit on by the end of the year. It assumes you can read a data sheet and now need the questions that decide whether the data sheet means anything.
Drop your details and we'll send MLOps and Model Serving: What Buyers Should Ask straight to your inbox - no spam, unsubscribe anytime.
Bring the serving contract you are about to sign and the worst hour your gateway recorded. A working session with our engineering leads, who will replay one against the other rather than present to you. SECTION 7 - FAQ - 5 to 8 questions
Talk to our engineers