framework

6 min read

MLOps and Model Serving Cost Calculator.

Inference is where AI spend settles. Training has an end date, a serving endpoint bills every hour it is up, and most teams still cannot say what one prediction or one million tokens costs them, or whether a hosted API would be cheaper than their own GPUs.

This calculator answers both with inputs you can check: traffic, measured throughput, a target utilisation and a GPU price. In its worked example, a 70B-class model on H100s, the hosted API stays cheaper until volume grows about 2.7 times.

That is one illustrative case rather than a benchmark, and the inputs that decide your answer are yours.

Download framework Talk to Our Engineers

In depth

The Cost of Serving a Model Is Set by Four Numbers.

Throughput per GPU is the number most often guessed.

It should be measured: your model, your engine and a replay of your real prompts, recorded at the point where your latency SLO still holds. Published benchmarks for the same engines disagree by tens of percent because their setups differ. Everything else in the calculation multiplies by this one figure. Measure it before you budget on it.

In shortMeasure it before you budget on it

A fleet sized for peak pays for the peak all day.

In the worked example, traffic averages around 10,400 tokens a second and peaks at 2.5 times that. A static fleet needs 15 GPUs, while autoscaling on load averages 8, which nearly halves the GPU bill and takes cost per 1M tokens from about $1.20 to $0.64. Autoscaling on queue depth or tokens in flight, rather than CPU, is usually the first saving. Idle capacity is the most expensive thing you run.

In shortIdle capacity is the most expensive thing you run

People decide build versus buy more often than GPUs do.

In the example, compute for self-hosting costs less than the API, but one platform engineer tips the monthly total the other way: about $36,700 self-hosted against $24,600 for the API. Break-even sits near 74 billion tokens a month, 2.7 times current volume. That is a useful answer rather than a failure. Stay on the API until the volume line is in sight.

In shortStay on the API until the volume line is in sight

Throughput gains move the break-even line further than price cuts do.

A 30% throughput improvement from tuning batching and prefix caching drops cost per 1M tokens from $0.64 to $0.48 in the example, and pulls break-even from 74 billion to about 46 billion tokens a month. FP8 quantisation, smaller or distilled models and routing easy requests to cheaper models work the same way. Each lever should be tested against quality on your own golden set. Cheaper tokens are only cheaper if the answers hold.

In shortCheaper tokens are only cheaper if the answers hold

The detail

Three Reasons to Self-Host That the Arithmetic Leaves Out.

Volume is the cleanest argument for running your own serving, but it is not the only one. Three others belong in the decision, each written down separately so that none of them is quietly priced at zero.

Zone · 01

Data that cannot leave

Where contracts, regulation or residency rules prevent prompts and outputs being processed by a third party, the API price is irrelevant. The question becomes whether a provider can run inside your cloud tenancy or whether you serve the model yourself. The artefact is a written data-handling requirement for each use case.

Zone · 02

A model no API offers

Fine-tuned, distilled or proprietary models have no list price per million tokens to compare against. Their cost is your serving cost, and the calculator becomes a sizing tool rather than a comparison. The artefact is a throughput benchmark of that exact model on your chosen engine.

Zone · 03

Latency and price you control

Dedicated capacity lets you hold a strict time-to-first-token target and a predictable monthly bill, while API prices and rate limits change at a provider's discretion. That control has a value worth naming. The artefact is the SLO the business has signed and the price exposure it is willing to carry.

By the numbers

The figures that make it a board-level conversation.

2.7x

volume growth needed before self-hosted serving beat a hosted API in the worked example

$2.82

per H100 GPU-hour on the neocloud rental index

About 1.5x

throughput gain from FP8 over BF16 for a 70B model on H100, with roughly half the memory

Inside the report

What you'll take away.

Step 1 - Measure throughput per GPU at your latency SLO

Replay real prompts on your model and engine and record total tokens per second per GPU where the SLO still holds. Do not borrow a vendor's number.

Step 2 - Record average and peak tokens per second

Pull a month of traffic and calculate both. The ratio between them tells you how much autoscaling can save.

Step 3 - Put cost per 1M tokens next to every model

Divide monthly serving cost by monthly tokens and review it with each model owner every month, alongside the API price it competes with.

Step 4 - Write down your break-even volume and review it quarterly

Fixed platform cost divided by the gap between the API price and your unit cost gives the line. Plan the move when traffic is on track to cross it.

Questions

Frequently asked.

Where do the example numbers come from?

They describe an illustrative 70B-class model in FP8 on H100s at 500,000 requests a day, with a $3.00 GPU-hour and a $0.90 per 1M token API price. They show how the method works. GPU and API prices change often, so use current quotes for your own run.

What GPU price should we use?

The blended rate you actually pay across on-demand, committed and spot capacity. Specialist clouds listed H100s at roughly $2 to $4.50 per GPU-hour in October 2026, and the Silicon Data rental index stood at $2.82 on 10 October. Hyperscaler on-demand rates are higher, and commitments bring them down.

Does this work for classical machine learning models?

Yes. Replace tokens with requests, and measure requests per second per replica instead of tokens per second per GPU. The same peak, utilisation, autoscaling and break-even logic applies, and many classical models will run on CPUs.

Why is target utilisation set at 70% rather than 100%?

Because bursts arrive faster than new capacity can start, and latency rises sharply as a GPU nears saturation. A target below full utilisation keeps the SLO intact during spikes. Set yours from load tests, not from the cost you would like to see.

The API came out cheaper. Should we stop self-hosting?

Not on cost alone. Data residency, fine-tuned models, latency guarantees and exposure to price changes can all justify self-hosting below break-even. Write those reasons down next to the arithmetic so the trade-off is explicit.

Which optimisation should we try first?

Autoscaling, if your fleet is static, because it changes the bill without touching the model. After that, quantisation, batching and prefix caching are usually low effort. Each should be validated against quality on your golden set before it reaches production.

How often should we rerun the model?

Monthly for cost per 1M tokens and quarterly for break-even. Rerun it straight away after any change to model, engine, hardware or traffic mix, since each one moves throughput per GPU.