Ordinary release engineering assumes the thing you deploy behaves the same way twice. A hosted model does not, and it can change underneath you without a commit landing anywhere. This reference holds twenty-three controls across seven families covering what that does to an otherwise normal release process: evaluation, gating, rollback, drift, cost, traceability and the day your provider retires the model. Every entry names the artefact that would prove it to somebody who did not build it, plus the failure mode it exists to catch. Open it at a family and implement that family the same week.
Three families sit downstream of the release and are the ones teams reach for late. They decide whether a feature can be shown to pay for itself, whether an incident closes on a record or on a hypothesis, and what happens on a date the provider chose.
Spend known monthly at the invoice and attributed to one platform line makes every feature look equally affordable, so none is ever switched off. Cost tokens, tool calls and retries per completed task and per tenant, emit the figure with the trace, and reconcile to the invoice monthly. A longer prompt or a deeper retrieval is a cost change, measured on the golden set in the same CI run as the quality gate.
The question in an incident is which version of what, on which context, for whom, and whether you can still show it six months later. A record holding the completion but not the resolved prompt or the retrieved chunk IDs closes on a hypothesis. Re-execute a stored run from its record, landing within a stated tolerance, and set retention tiers against your longest complaint window rather than against storage cost.
Providers retire models on their own schedule, so end-of-life is the one change every team knows is coming and almost none has budgeted for. List every model, embedding model and adapter with its announced end date and notice period, and treat a blank date as a finding in itself. Run the golden set against a successor quarterly, before any notice arrives, and price the embedding swap as the full re-index it is.
Cases taken from production rather than imagined, each carrying the expected outcome and the error class it guards, versioned with a changelog entry per release. The suite is built before the feature it tests.
Percentages, a minimum dwell time at each step, and the figures that halt the rollout, all agreed in advance and read by the pipeline itself. A ramp that needs a human to notice is not a gate.
Refuse moving aliases at build time, pin a dated version, and hold one alternative you have run against the same golden set inside the quarter. Provider independence is a configuration decision made early or a scramble made late.
One accountable engineer per alert with a stated first action, and a synthetic breach fired quarterly to time the page and the acknowledgement. An escalation path that has never been exercised is a document, not a control.
Seven families covering what a non-deterministic dependency does to it: an evaluation harness, gating and progressive rollout, rollback and restore, drift and quality monitoring, cost and rate governance, incident traceability and retention, and model end-of-life. Twenty-three controls in total, each mapped to one of the four gates in the production standard.
One pass through staging validates a deterministic service and tells you very little about this one. The ramp is where the evidence comes from: shadow evaluation with no side effects, an untreated holdout kept on the previous version, and abort thresholds the pipeline reads for itself. A canary watched only on latency and error rate ships a quality regression to everybody.
Only if the thing you roll back to still exists. A restore point has to pin code, prompt, retrieval configuration, tool schemas and the model reference together at stated prior values, written by the pipeline rather than picked as a commit. A commit hash on its own restores one of the five and leaves the other four wherever they drifted to.
Per completed task, not per month. Tokens, tool calls and retries costed per task and per tenant, emitted with the trace rather than derived afterwards, then reconciled to the provider invoice monthly. Add a cost delta per release measured on the golden set against a stated ceiling, so a wider context window shows up before it ships.
Set tiers rather than one number: full trace, sampled trace and aggregate, each given a stated number of days chosen against the longest complaint window you face rather than against storage cost. Days are counted from the moment the record is written. The test is a support query resolving a complaint from last quarter to its exact run, timed.
Not in this catalogue. Production AI Under Regulation handles that side: the duties that begin at go-live and run for the life of the system, and the point at which a change is substantial enough to reopen conformity without anyone in the release path noticing. Read them side by side when an auditor wants the control and the duty behind it.
PRD-1. Nothing else in the catalogue means much without a suite that can tell you a change made things worse, and the gating, drift and end-of-life families all call into it. After that, the restore point, since pinning five values as one unit is a line of pipeline code early and a re-architecture on a system already serving traffic.
Drop your details and we'll send Production AI: An Engineering Reference straight to your inbox - no spam, unsubscribe anytime.
A golden set running in your CI, a pinned restore point tested rather than assumed, a timed rollback drill and an architecture note on traceability. Working code in your repository, and everything produced is yours to keep. SECTION 7 - FAQ - 5 to 8 questions
Book a production readiness review