framework

5 min read

MLOps and Model Serving Operations Runbook.

When a model endpoint degrades, the damage is immediate: slower products, wrong decisions, GPUs still billing and engineers pulled off roadmap work. Most of that cost builds up in the first half hour, while people work out what is happening and who is allowed to act.

This runbook removes the delay. It defines what to measure, how severe each situation is and who does what, then gives first actions for seven failure modes, from latency breaches and out-of-memory crashes to drift and cost spikes.

It is written to be copied into your incident tool and adapted, not read once and filed.

Download framework Talk to Our Engineers

In depth

Why Model Incidents Run Longer Than They Should.

Waiting for permission is the most expensive minute of most incidents.

When a release goes bad, the fastest fix is nearly always to roll back, yet on-call engineers often hesitate because nobody told them they could. The runbook pre-approves rollback of any model release by whoever is on call, with the decision reviewed afterwards. Rule one is that rollback never waits.

In shortRule one is that rollback never waits

Model incidents hide inside healthy dashboards.

An endpoint can answer every request inside its latency target while the model behind it has drifted for weeks. The runbook sets signals at four layers, service, capacity, model and cost, each with an example threshold and a named recipient, so drift goes to the model owner and cost spikes go to the owner and FinOps. An alert without an owner is noise.

In shortAn alert without an owner is noise

LLM serving fails in ways web services do not.

Time to first token rises when queues build, GPUs run out of memory on long inputs, throughput collapses when the traffic mix shifts and runaway agents can multiply cost overnight. Each failure mode has its own playbook with a trigger, a 15-minute triage, mitigations and follow-ups. Generic incident processes do not cover them.

In shortGeneric incident processes do not cover them

Every incident should leave a guardrail behind.

The follow-up in each playbook is a change rather than a report: a new alert on queue depth, an input-length limit at the gateway, a failing case added to the golden set. A post-incident review that ends without a new test or alert has not finished. The second occurrence is the one that should never happen.

In shortThe second occurrence is the one that should never h…

The detail

Three Habits That Keep the Pager Quiet.

Playbooks shorten incidents, and routine operations prevent them. Three habits do most of the preventing, and each can be checked in a calendar.

Zone · 01

A weekly model review

Model owners review drift and quality trends for each model every week and triage early-warning tickets before they become pages. Degradation caught in a weekly review is a retraining task rather than an incident. The artefact is a short weekly note per model, with the trend charts attached.

Zone · 02

A monthly cost review

Cost per 1M tokens or per 1K predictions is reviewed by model every month, with replicas and GPU types right-sized against actual load. Spend that crosses budget is treated as an incident rather than discovered at month end. The artefact is a cost-per-model table with an owner beside each line.

Zone · 03

A quarterly drill

Once a quarter, the team runs a timed rollback and a failover, reviews the runbook and retires endpoints nobody uses. A rollback that has never been practised will not take five minutes when it matters. The artefact is a drill log with the measured times and the fixes that followed.

By the numbers

The figures that make it a board-level conversation.

Under 5 min

the rollback target the runbook pre-approves on-call engineers to act on

91%

of model and dataset pairs tested showed quality degradation over time

27%

of large enterprises have implemented automated GPU resource-sharing dashboards

Inside the report

What you'll take away.

Step 1 - Pre-approve rollback for whoever is on call

Write it into the incident policy this week. No model rollback should wait for a manager or for the model's author.

Step 2 - Put the seven playbooks into your incident tool

Replace the placeholders with your dashboards, rollback commands and escalation contacts, so responders can act in minute one.

Step 3 - Give every alert an owner

Route service and capacity alerts to on-call, drift and quality to model owners, and cost to owners and FinOps. Delete the alerts nobody owns.

Step 4 - Run a 30-minute rollback drill this month

Time a real rollback of one model, fix whatever made it take longer than five minutes, then repeat the drill every quarter.

Questions

Frequently asked.

How is this different from a standard SRE runbook?

It keeps the SRE foundations of golden signals and severities and adds what model serving needs: time to first token and time per output token, GPU memory and queue depth, drift and live quality, and cost per 1M tokens.

Several of the seven playbooks have no equivalent in a typical web service runbook.

Which SLOs should we start with?

Availability, latency and, for LLMs, time to first token. The runbook gives example thresholds, such as 99.9% availability and a time-to-first-token p95 under one second, but yours should come from load tests and the first two weeks of production data.

Who should be on call for a model?

Someone who can operate the model, not only the cluster. Platform on-call handles infrastructure, scaling and rollback, and the model owner judges quality and approves fallbacks. Both roles need to be reachable for SEV1 incidents.

Why is a cost spike treated as an incident?

Because runaway agents, retry loops and missing scale-down policies can multiply a serving bill within days. A cost alert at 120% of budget run-rate, with per-caller attribution, catches the problem while it is still small.

What should a drift alert trigger?

First, a check of upstream data, since broken pipelines cause many apparent drift events. If the inputs really have changed, the model owner informs the business owner and starts retraining on recent data, evaluated against the golden set before release.

How often should the runbook change?

After every incident and at every quarterly review. Each post-incident review should name the playbook it updated. A runbook that has not changed in six months is probably out of date.

Does it cover failover between hosted model providers?

Yes. Playbook seven covers provider and region outages: confirm the outage, fail over to a secondary model or region through the gateway, and degrade non-critical features. Test failover every quarter and record any quality difference in the fallback model.