AI vendor assessment fails the same way at mid-market scale. The questionnaire is long, the vendor answers it in marketing language, nobody has time to verify anything, and the assessment becomes a filed document rather than a decision input. Its weakest answer then surfaces eighteen months later in your own customer's review. This scorecard resists that in three ways: knockouts first so a bad fit ends in an hour, a stated zero and four for every criterion so a vague answer becomes hard to score, and a question bank phrased so the evasive answer is visibly evasive.
The difference between a scorecard and a questionnaire is mostly in how it is structured. Three choices carry that difference.
Training on customer data with no opt-out on your plan. Refusal to identify sub-processors. No data processing agreement, or one that carves out AI processing. Inability to state where data is processed. No breach notification commitment, or one measured in reasonable time. Each is irreversible or unenforceable rather than merely weak, and each is often a signal you are on the wrong plan rather than with the wrong vendor.
On retention, zero is indefinite or as long as necessary and four is a defined maximum, configurable, with zero-retention available for sensitive workloads. On model change, zero is no changelog and four is a published changelog covering behaviour rather than only features. Making the endpoints explicit stops a scorer crediting a vendor for what their category usually does well.
Ask what the three things this model does worst in your use case are. Deflection here is the most informative response in the bank, because a vendor who cannot name limitations either does not know them or will not state them, and both tell you what their documentation is worth when your own customer asks.
Training on your data, sub-processor transparency, a DPA scoped to AI processing, processing locations, and a breach notification period in hours or days.
Fifteen questions phrased so an evasive answer is visibly evasive, each paired with what a good answer actually contains: a number, a setting name, a clause, a location.
Score what you can see. A screenshot of the training setting on your own account beats a clause, and the gap between the two is a finding in itself.
Eighteen terms from a training prohibition through to deletion of derived artifacts on exit, plus the six triggers that should force a reassessment later.
Start with five. Is customer data used to train models available to other customers, and on which plans. How long is data retained, where, and can we configure zero retention. Under what circumstances can staff read our inputs. What exactly is deleted on termination, including embeddings and fine-tuned artifacts. And what is the contractual breach notification period.
Generally yes, under conditions now standard on business and enterprise tiers: provider training disabled and verified in the console rather than assumed from the contract, a DPA covering the AI processing, bounded retention, sub-processors enumerated, and sensitive classes blocked at your application layer.
Beyond standard terms: a prohibition on training with your data, output ownership with the vendor asserting no rights, IP indemnity for outputs, model version pinning or a deprecation notice period, notice of material change with a termination right, deletion on exit covering derived artifacts, and flow-down permission for your own customer commitments.
Annually as a floor, and at renewal when your leverage peaks. Also on a material model change or deprecation, on any vendor security incident whether or not they say it affected you, when you first send a new data class, and when a customer contract imposes a new flow-down.
No, it is an honest and useful answer, and roughly one in five knockout failures is exactly this. You are on the wrong plan rather than with the wrong vendor, which makes it a procurement question. The genuinely bad answers are the ones that restate the question as a commitment to taking security seriously.
Yes, and they should stay separate, because different people at the vendor answer them and different people at yours verify them. This covers governance, contractual and transparency posture. The AI Security Vendor Scorecard covers how the system is actually built and includes an architecture interview a questionnaire cannot substitute for.
Drop your details and we'll send AI Governance Vendor Scorecard straight to your inbox - no spam, unsubscribe anytime.
Which makes it a procurement conversation rather than a replacement. Work through your largest AI vendor with our engineering leads. A working session, not a sales pitch. SECTION 7 - FAQ - 5 to 8 questions
Talk to our engineers