framework

5 min read

MLOps and Model Serving Decision Framework.

Model serving looks like a tooling problem, but cost, speed and risk are mostly fixed by six early decisions: whether to build or buy, which serving pattern, which platform, which inference engine, which hardware and how to release.

Made by habit, each one hands you someone else's trade-offs for years. AWS has put inference at up to 90% of production ML compute, so these choices shape the largest line on the AI bill.

This framework gives each decision explicit criteria, a default answer and the conditions that override it, plus a record that keeps the reasoning after the people who made it have moved on.

Download framework Talk to Our Engineers

In depth

Six Choices Made Early That Set the Bill for Years.

Start on an API and earn the right to self-host.

A hosted model is the fastest way to learn what your real traffic looks like. Move to self-hosting when one of four conditions holds: an API model fails your golden set, data cannot leave your environment, volume passes your break-even, or you need latency the provider will not guarantee. Design the gateway so the switch is a routing change. A rewrite is the expensive way to change your mind.

In shortA rewrite is the expensive way to change your mind

Serve at the latency the business needs, not the latency engineering prefers.

Results needed within a day belong in batch on spot capacity. Per-request decisions on structured data run well on CPUs at 10 to 100 milliseconds, generated text needs streaming LLM serving, and offline or strict-privacy use cases may belong on the device. Every endpoint that can move to batch is a GPU that can be switched off.

In shortEvery endpoint that can move to batch is a GPU that…

Platform choice is a statement about your team, not your ambition.

Open-source on Kubernetes gives the lowest unit cost at scale, but only with engineers who can run GPUs in production. Hyperscaler platforms suit teams standardised on one cloud who value speed, and specialist inference providers suit spiky LLM traffic that has to launch this quarter. Weight the criteria before scoring, not after. Weights set afterwards only confirm a preference.

In shortWeights set afterwards only confirm a preference

Compare engines and hardware on cost per million tokens, not headline speed.

vLLM, SGLang, TensorRT-LLM and Triton each lead on some workloads, and published benchmarks conflict because their setups differ. A more expensive GPU that serves three times the tokens is the cheaper choice. Run a two-day bake-off on your own prompts at your latency SLO. The only benchmark that transfers is the one run on your traffic.

In shortThe only benchmark that transfers is the one run on…

The detail

Three Habits That Keep the Decisions Honest.

A framework is only as good as the discipline around it. Three habits stop the six decisions drifting back into preference, and each leaves something behind that can be checked.

Zone · 01

Score before you shortlist

Agree weights for time to production, unit cost, team capacity, data fit, portability, LLM performance and existing tooling before any vendor is in the room, because vendors push the criteria they win on. The artefact is a signed weighting table, dated before the first demo.

Zone · 02

Benchmark on your own traffic

Same model, same precision and a replay of real prompts, measured at your latency SLO. Two days of benchmarking settles arguments that months of reading cannot, and the number feeds straight into cost per 1M tokens. The artefact is a benchmark report with engine, version and settings recorded.

Zone · 03

Write the exit plan first

For every decision, record what it would take to reverse it and which change in volume, price or quality would reopen it. Decisions without review triggers become permanent by default. The artefact is a one-page decision record with an owner and a review date.

By the numbers

The figures that make it a board-level conversation.

Up to 90%

of production ML compute cost goes on inference rather than training

44%

of large enterprises still assign GPU workloads manually or have no utilisation strategy

About 1.5x

throughput gain from FP8 over BF16 for a 70B model on H100, with roughly half the memory

Inside the report

What you'll take away.

Step 1 - List the six decisions behind your current serving stack

Write down what was chosen and who chose it. Any gap in that list is a decision that was made by default.

Step 2 - Move every endpoint that can tolerate delay to batch

Ask each business owner the latency their use case really needs, and switch off the GPUs that the answers free up.

Step 3 - Run a two-day engine bake-off before the next platform choice

Measure throughput at your SLO on real prompts for at least two engines and convert the results to cost per 1M tokens.

Step 4 - Record each decision with its exit plan and review trigger

Use the decision record so the reasoning, cost impact and reversal cost survive team changes.

Questions

Frequently asked.

Which decision should we make first?

Build or buy, because it determines whether the others are yours to make. If a hosted API meets quality, data and latency needs below your break-even volume, platform, engine and hardware become the provider's problem, and your effort goes into the gateway and evaluation.

Is there a best inference engine?

Not in general. vLLM is a strong default with broad model support and no compile step. SGLang tends to lead when prompts share long prefixes or need structured output. TensorRT-LLM can be fastest on NVIDIA hardware once tuned, but its engines are tied to a specific GPU type and count.

Should we use Kubernetes for model serving?

If you have engineers who already run it in production and enough volume to justify them, yes. KServe became a CNCF incubating project in 2025, and llm-d adds distributed LLM serving on top. Without that team, a managed or specialist platform usually ships faster and costs less overall.

How should we compare GPUs?

On cost per 1M tokens at your latency SLO. Price per hour is misleading on its own, because throughput differs widely between GPU generations and precisions. FP8 on H100, for example, delivered about 1.5 times the throughput of BF16 in NVIDIA's published 70B results.

What release strategy should a new model use?

Match it to the risk of a bad model. Blue-green suits low-risk internal tools, canary releases suit customer-facing models with quick feedback, shadow then canary with human review suits money or safety decisions, and A/B testing suits changes whose effect on the KPI is unknown.

How often should these decisions be revisited?

When a review trigger fires, such as volume crossing break-even, a large price change or a new model generation, and at least once a year otherwise. The decision record names the trigger so nobody has to remember it.

Who should own the decisions?

The platform lead for engine, hardware and platform, the model owner for pattern and release strategy, and both together for build or buy. Each decision record names one accountable owner.