Nobody buys an accelerator hour. They buy the share of it that does work somebody asked for, and that share is the term no serving budget measures. The rate gets negotiated for three months; the denominator underneath it is never instrumented.
This paper prices a shared serving platform as infrastructure amortised across the teams on it, sets that against the version where every team builds a thin copy, and runs the sensitivity both ways.
Utilisation turns out to be worth about three times what the unit price is worth.
Six cost lines sit under a serving platform and a typical plan watches the first three, since those are the ones that arrive as invoices. The other three decide whether the first three were a good price, and two of them never appear on a bill at all.
Not an invoice line, and the denominator under every other one. Per-team reservations, cold start padding and workloads that never share a scheduler all push it up. Reserve held deliberately for burst and node failure is bought capability; capacity stranded because an interactive daytime load and an overnight batch load were funded separately is waste wearing the word headroom.
Priced per deprecation event in evaluation and re-tuning effort, and multiplied by the number of independent stacks that each have to repeat it. It recurs, it is real engineering, and it appears in approximately no serving model. Carry it as a standing reserve for two events a year rather than meeting it as a surprise.
Per model and per tenant: traces, usage records and the chargeback data that lets a team be shown what it actually consumed. Added after the fact it produces an argument instead of a figure, and the utilisation term the whole model turns on cannot be produced at all. The tenant identifier usually goes missing before the first span is written.
The share of paid accelerator time doing work a user asked for is the number this model turns on, and most platforms cannot produce it. Instrument it before the next procurement call.
Fund serving capacity on a line of its own, then attribute consumption back from usage records. A shared gateway paid for out of one product's budget is sized by one product's forecast, and nothing in that forecast knows about team four.
Checkpoint deprecations land on someone else's schedule. Price two a year, hold the evaluation suites in one place, and the event becomes a planned run instead of nine separate fire drills.
Ask what concurrency and batch size produced it, then run the bake-off against the traffic shape you actually serve rather than a clean synthetic one. Sixteen streams in flight serve eleven times the volume of one.
It is a ratio whose denominator nobody stated. The same model on the same hardware serves a very different volume per hour depending on how many requests are in flight together, so the figure carries a concurrency assumption inside it, and that assumption is usually the vendor's demo rather than your traffic.
Less than the negotiation costs, on the model's own arithmetic. Swinging the capacity price by a fifth either way moves the unit cost by 14%, while running utilisation across the range a real platform occupies moves it nearly three times as far. The second lever sits entirely inside your own engineering.
It is ordinary, and it means the real cost of a working hour is more than three times the headline one. The useful question is how the gap splits. Reserve held deliberately for burst and node failure is bought capability.
Capacity stranded because two workloads with opposite daily shapes never share a scheduler is not.
In the worked example, about $2.2M a year and $6.6M over three, for identical useful work. Nine separate peaks need 36 fleet units where one aggregated peak needs 24, which drags utilisation from thirty per cent to twenty, and nine registries and gateways get paid for instead of one. The inputs are illustrative.
Modelled, and deliberately so. Every input in the worked example is illustrative, chosen to show the shape of the arithmetic rather than to report an observed platform, and the paper quotes no vendor rate for an accelerator or a token anywhere.
Replace all of them with your own measurements before the number leaves the room.
With utilisation, and with the tenant identifier that makes it computable. If you want a view of the platform's condition alongside its cost, Benchmarking MLOps and Model Serving scores one shared platform out of a hundred across four domains, and tenancy and cost attribution is the domain that usually comes out lowest.
A VP of platform and a finance partner, in the same room. The engineering half owns scheduling, routing and release code, which is where utilisation is actually decided, and the finance half owns the line items the engineering half never sees. Neither can fix the bill alone.
Drop your details and we'll send The Economics of MLOps and Model Serving straight to your inbox - no spam, unsubscribe anytime.
Bring your fleet size, your team count and whatever your serving tier can tell you about utilisation. Our platform engineers will run the model on your measurements rather than ours, and tell you which lever is worth pulling first. SECTION 7 - FAQ - 5 to 8 questions
Book a cost review