You are not buying the demo. You are buying what it costs to keep that demo awake at three in the morning, on traffic nobody rehearsed. The usual evaluation fails in three predictable ways, and at least as often it is the buyer's failure rather than the seller's: a prototype judged on curated inputs, a headline success rate that hides its distribution, and a contract drafted for software that does the same thing twice. These eight question sets fix that.
Vendors offer accuracy percentages that sound like commitments. They are not enforceable in any practical sense, since the two of you will argue about the test set every time one is invoked. Here is what to put in the agreement instead.
Determinism is real here, so a number means something and can be measured by both sides without an argument. Put the uptime target, the response time percentiles and the service credit in the agreement, and keep them separate from anything about output quality. This is the part of a conventional SLA that still works unchanged.
Output quality belongs in a reporting obligation rather than a penalty clause. Name the set: task success on an agreed evaluation set, groundedness, refusal and escalation rates, and drift against a baseline, reported monthly and auditable. Attach a remediation path to a breach instead of a service credit, since the credit never covers what a bad answer costs you.
The clause almost nobody has, and the cheapest to ask for while a deal is still being negotiated. You want a written notice period before the model underneath you moves, a pinned version you may stay on for a stated window, and the evaluation delta published with the change. Who re-runs your evaluation set matters as much as the notice itself.
Pick the month with the strangest queries, the broken documents and the cases your support team complains about. A pilot on a tidy sample only ever measures the sample.
Define the categories of wrong answer and the severity of each while nobody yet knows the outcome. Once results land, every definition turns into a negotiation you will lose.
Ask for the notice period, the pinned version window and the published evaluation delta now. How a vendor reacts to that clause tells you more than the pilot result will.
Evaluation sets, prompts, retrieval configuration, fine-tune recipes and traces land in your repository as they are produced, not in a handover deck during the final week.
Ask which failures they are carrying in production right now. An answer of none means nobody has looked. A good answer is a named list with severities and owners, plus the evaluation case in continuous integration that catches each one.
No. It is unenforceable in any practical sense, since you will disagree about the test set every time it is invoked, and it crowds out commitments that would be useful. Take availability and latency as hard numbers, and output quality as a measured metric with a remediation path.
In most shops, the previous container. Prompts, retrieval configuration and model version are not inside that artefact, so the rollback restores the code and leaves the behaviour where it was. Ask for one versioned release unit holding all four together, and ask when they last rehearsed it.
Ask whether you can run the system without them for a month, starting now, and make them name the month. A partner who cannot be stood down for thirty days has transferred nothing yet, whatever the handover deck says.
Not in a straight line, so distrust a projection drawn from this month's invoice. Ask for a written account of what is fixed, what scales with requests, and what scales with context length and retries, naming where the curve bends. Then instrument cost per request from the pilot's first week.
Every answer should map to a control in Production AI: An Engineering Reference, our catalogue of numbered controls and the artefact that proves each one. An answer with nowhere to land in it is marketing. Use this playbook to ask, and that reference to check.
CTOs and heads of engineering about to sign with an integrator who will write the code, a managed platform that will run the inference, or a product they will embed and then depend on. It assumes the decision is close and the demo already went well.
Drop your details and we'll send Production AI: What Buyers Should Ask straight to your inbox - no spam, unsubscribe anytime.
Put us beside the vendor you are evaluating and score both against the same eight questions. Ninety minutes with the engineers who would do the work, and no deck. SECTION 7 - FAQ - 5 to 8 questions
Book a vendor review