framework

6 min read

MLOps and Model Serving Reference Blueprint.

Most organisations do not have a modelling problem. They have a delivery problem: a working model waits months for packaging, serving, monitoring and on-call, and Gartner surveys between 2022 and 2025 put the share of AI prototypes that ever reach production at 41 to 54%.

Once a model is live, inference keeps billing every hour, and AWS has put inference at up to 90% of production ML compute. This blueprint sets out the platform that closes that gap: six layers, one golden path and a 90-day build in which every phase delivers something measurable.

It is a target architecture rather than a product recommendation, and the trade-offs between open-source, hyperscaler and specialist platforms are laid out rather than settled for you.

Download framework Talk to Our Engineers

In depth

Why Good Models Stall Between the Notebook and the Endpoint.

Every model rebuilt from scratch is a project, not a release.

When there is no shared path to production, each team rebuilds packaging, serving, monitoring and rollback for its own model, and the second model takes as long as the first. That is how a few months of modelling becomes an eight-month delivery programme. A golden path reverses the economics: the platform is paid for once, and every later model rides on it in days rather than quarters. The platform is the multiplier, not the model.

In shortThe platform is the multiplier, not the model

Serving cost is decided by architecture long before the first invoice.

A model that feeds a nightly report does not need a GPU endpoint running around the clock, yet static fleets sized for peak traffic are the norm. With inference accounting for up to 90% of production ML compute, the choice between batch, online, streaming and edge serving is a budget decision as much as a technical one. Matching the pattern to the latency the business actually needs is usually the largest saving available. The cheapest GPU is the one you switch off.

In shortThe cheapest GPU is the one you switch off

A healthy endpoint can serve a model that has been wrong for weeks.

Service monitoring tells you the server is up, and it says nothing about whether the predictions are still good. In one study, 91% of model and dataset pairs showed measurable degradation over time. Drift, live quality and cost per request belong on the same dashboard as latency and errors. If quality is not monitored, it is not managed.

In shortIf quality is not monitored, it is not managed

Lock-in is cheapest to avoid before the first model ships.

Open-source stacks on Kubernetes, hyperscaler platforms and specialist inference providers each win somewhere, and most enterprises end up running a mix. The blueprint keeps the interfaces between layers stable, so an engine, registry or provider can be swapped without rebuilding the platform. That only works if it is designed in from the first deployment. Portability is a design choice, not a migration project.

In shortnot a migration project

The detail

Three Layers That Decide Whether the Platform Pays Back.

The blueprint has six layers, but three carry most of the value. Each one turns a recurring cost into a one-off build, and each can be checked against a single artefact.

Zone · 01

The golden path

Eight gated steps take a model from commit to full traffic: commit, train, evaluate, register, package, shadow, canary and promote. Every gate protects something specific, from reproducibility to revenue at risk, and the previous version stays warm so rollback takes minutes. The artefact is a release pipeline that a new model can use without new code.

Zone · 02

The serving layer

A gateway handles authentication, quotas and routing, engines such as vLLM, SGLang or TensorRT-LLM do the work, and autoscaling runs on queue depth or tokens in flight rather than CPU. This is where GPU utilisation, and so unit cost, is won or lost. The artefact is a cost per 1M tokens or per 1K predictions for every model.

Zone · 03

The observe layer

Service health, model health and cost are measured together, per model and per version. Drift and quality monitors alert with the same discipline as latency, and each request can be traced to the model, prompt and configuration that produced it. The artefact is one dashboard per production model, with an owner's name on it.

By the numbers

The figures that make it a board-level conversation.

41 to 54%

of AI prototypes reach production, across Gartner surveys from 2022 to 2025

Up to 90%

of production ML compute cost goes on inference rather than training

91%

of model and dataset pairs tested showed quality degradation over time

Inside the report

What you'll take away.

Step 1 - Map your current stack onto the six layers

Name a tool and an owner for each layer. Any layer with neither is where the next incident or the next six-month delay will start.

Step 2 - Choose the first two models for the golden path

Pick the two with the clearest business KPI and the highest serving cost, and build the platform around them rather than in the abstract.

Step 3 - Match every endpoint to the latency it really needs

Ask each business owner what response time the use case requires. Every endpoint that can move to batch is a GPU you can switch off.

Step 4 - Publish before-and-after numbers at day 90

Report release time, rollback time and cost per prediction for both models. Those three figures fund the next phase.

Questions

Frequently asked.

We already use a managed ML platform. Do we need this blueprint?

Probably, as a checklist rather than a rebuild. Managed platforms cover several layers well, but serving efficiency, cost per prediction and model-quality monitoring are often left at their defaults. The blueprint shows which layers your platform already covers and which still depend on individual teams.

Is this only for LLMs?

No. The six layers, the golden path and the release gates apply to classical models on structured data as much as to LLMs. The serving layer differs most: classical models usually run on CPUs or small GPUs, while LLM serving is GPU-bound and priced per token.

Should we standardise on open-source or a hyperscaler?

It depends on volume, team skills and data constraints. Open-source on Kubernetes gives the lowest unit cost at scale but needs platform engineers. Hyperscaler platforms trade a compute premium for speed, and specialist inference providers suit fast LLM launches. Most enterprises end up with a mix.

Which inference engine does the blueprint assume?

None by default. vLLM is the broad general-purpose choice, SGLang tends to lead on long shared prefixes and structured output, and TensorRT-LLM can deliver the highest throughput on NVIDIA hardware once tuned. Benchmark on your own traffic before committing.

How long does the build take?

The plan runs over 90 days in three phases: foundation, safe delivery, and efficiency and scale. Each phase exits with a measurable result, such as the first model live on the path, rollback under five minutes, or a second model shipped in under ten working days.

How does this relate to MLOps maturity models?

It maps onto them. The 90-day plan takes most organisations from manual deployment to automated delivery, roughly Google's level 2 or Microsoft's level 3, for their first two models. Full automation follows as more models move onto the path.

What should we measure to prove the platform works?

Time from approved model to live traffic, rollback time, GPU utilisation, cost per prediction, the share of production models with drift monitoring and time to restore service. Baseline them before you start so the improvement is visible.