Static analysis reads code, dependency scanners read manifests, and neither reads the retrieved document that tells your agent to email the customer list outside the company. This report sets out the five failure classes that cover almost everything exploited in production, and the four design decisions that survive a red team.
The trap most engineering teams walked into: ship the AI feature against a demo date, wire a broad service account into retrieval, hand the agent one shared credential with write access it never needs, and put a prompt-injection classifier at the input as though there were a grammar of malicious instructions to match.
What the teams surviving a red team do: assume the model will be persuaded and bound what it can reach: a privileged planner that never sees untrusted content, one credential per tool scoped to a single action, retrieval authorisation enforced in the query, and schema validation at every consuming boundary.
Not Filters
Split the pipeline. A privileged planner that never sees untrusted content, and an unprivileged worker that reads it but cannot act. Input filtering is not a control here: there is no grammar of malicious instructions to match, and every blocklist is one paraphrase from bypass. Treat classifier detection as telemetry.
One credential per tool, scoped to the narrowest action the use case needs, so a read agent cannot write and a support agent cannot touch billing. Then put a human in the loop on anything irreversible: payments, deletions, external messages, code merges, permission changes. Test the boundary by attempting what should be impossible.
An index built once by a service account that reads everything will return the board pack to a contractor who asks the right question. Enforce permissions as a pre-filter on the candidate set, because filtering unauthorized documents out after retrieval is a leak waiting for a ranking bug. Add per-role tests on every corpus change.
Model, version, data sources, tools, credentials and owner for each one. Shadow AI costs roughly $670,000 more per breach precisely because it is invisible, and nobody can scope an incident on a system that appears in no register.
No shared keys and no write tokens where read suffices. Then treat every completion as hostile input to whatever runs next: schema-validate on the way out, parameterise queries, encode contextually and allowlist egress, exactly as you would for public internet traffic.
Token flooding, adversarially expensive inputs, recursive agent loops and retry storms against a metered upstream all convert traffic into invoice. Per-tenant budget ceilings, a concurrency cap, a maximum tool-call depth and a breaker that fails closed with a defined fallback.
A versioned suite of injection, leakage and exfiltration cases written against your specific tools and permissions, failing the build when it regresses. Add an external OWASP-aligned exercise before launch, and keep a model bill of materials with pinned digests rather than tags.
Classifiers cannot do that job. There is no grammar of malicious instructions to match, and payloads can be encoded, translated, split across documents or hidden in white text. Treat them as useful telemetry and a reasonable speed bump. Deploy one as a boundary and a red team will get through it in an afternoon.
Indirect injection does not need your interface. Any content the model ingests is a delivery path: an inbound email, a supplier PDF, a customer support ticket, a scraped page in the corpus. Internal deployment removes the login page, not the ingestion path.
Usually not the embeddings, but yes to the metadata and the query path. You need source-system permission identifiers stored alongside each chunk and applied as a pre-filter before ranking. Budget a sprint for the plumbing, plus per-role tests that keep it honest afterwards.
Standard scoping tests your code and infrastructure. It does not attempt indirect injection through the corpus, exfiltration through tool chains, embedding inversion or cost exhaustion. Those need adversarial cases written
against your specific tools and permissions, then kept in CI as the system changes.
Tool scoping and output validation cannot be deferred cheaply, because both are wired through the call path. Inventory, logging classification and CI evaluation can follow within a quarter. The pattern in IBM's data is unpriced
deferral: 97% of breached AI systems had no AI-specific access controls.
Drop your details and we'll send The AI Attack Surface Your Application Security Scanner Cannot See straight to your inbox - no spam, unsubscribe anytime.
In a two-week trial sprint we deliver a threat model for one live AI feature and an adversarial evaluation suite running in your CI.
Book a security review