Teams are wiring LLMs and agents into tickets, CRMs, code and payments faster than they are deciding what those agents are allowed to touch. In IBM's 2025 study, 97% of organizations that reported an AI-related breach said they lacked proper AI access controls, and 63% had no AI governance policy.
AI security is the practice of protecting models, prompts, retrieval data, agents and tools from attack and misuse, and this handbook maps it to OWASP, MITRE ATLAS and NIST. The evidence is breach surveys, analyst predictions, public frameworks, one peer-reviewed jailbreak study and disclosed incidents.
It describes where attacks land and which controls limit them, not how secure any particular deployment is, and the Gartner figures are forecasts rather than measurements.
None of this requires a new security discipline. It requires treating the LLM as an untrusted, highly capable user inside the system and checking that the architecture assumes as much. Three questions test that, each answerable with one named artifact a reviewer can read.
the model or code
Ask where authorization happens when the model proposes an action. The answer should be deterministic code outside the model that checks the requesting user's rights, with human approval for payments, deletions, external email and permission changes. If the model's own judgment is the only gate, one successful injection reaches everything the agent can reach. The artifact is an agent permission table that lists each action type, its default autonomy and its required control.
Ask what data and tools a fully compromised agent could touch. Retrieval should filter by the caller's identity at query time, indexes should be partitioned per tenant, and no single flow should combine private data, untrusted content and an outbound channel. Anything wider is blast radius the team has accepted without saying so. The artifact is a data flow diagram with every untrusted input and every triggerable action marked.
Ask for the attack success rate per threat category, measured with realistic retry budgets. A team running PyRIT, garak or promptfoo in CI on every prompt, model or tool change can answer in numbers, while a team relying on provider defaults can only say the filter is switched on. Because success climbs with attempts, a one-shot result is not an answer. The artifact is a baseline ASR report with release thresholds attached.
Draw the data flow, mark every point where untrusted text enters the context window, list each triggerable action and assign one preventive control, one detective control and one owner per path.
Give each agent its own workload identity, on-behalf-of tokens and short-lived credentials, allowlist tools and arguments, and require human approval for irreversible, financial or privilege-changing actions.
Pair a provider filter, a classifier on retrieved content and deterministic policy code for tool calls, then track attack success rate per category with realistic retry budgets and set release thresholds.
Rate eight domains from 1 to 4 on the scorecard, treat a total under 16 as not ready for broad release, and tag controls for NIST AI RMF, SOC 2 and ISO/IEC 42001.
AI security is the protection of AI systems, including models, prompts, training and retrieval data, agents and tools, against attacks and misuse. For production LLM apps it focuses on prompt injection, data leakage, unsafe outputs, poisoned supply chains and over-privileged agents.
It builds on standard application security and is distinct from using AI to run security operations, which this handbook does not cover.
Prompt injection is crafted text, typed by a user or hidden in a document, email or web page, that overrides an LLM's intended instructions. It is LLM01 on OWASP's 2025 list, and no filter fully prevents it.
Reduce the impact by keeping authorization outside the model, scoping data access to the user, filtering inputs, stripping markdown images and hidden links from outputs, and requiring human approval for high-impact actions.
Attackers use role-play framing, many-shot prompts full of fake compliant dialogues, encoding tricks, multi-turn escalation and Best-of-N random variations. Best-of-N reached 89% success on GPT-4o at 10,000 attempts. Defend with layered classifiers, per-user rate limits and retry budgets, continuous red teaming, and an architecture that limits what a jailbroken model can reach, since some attempts will get through.
Give each agent its own workload identity and use on-behalf-of tokens so it acts with the requesting user's rights. Issue short-lived credentials from a secrets manager, allowlist tools and arguments such as refund caps, and keep agents out of production write paths by default.
Require human approval for payments and deletions, and avoid flows that combine private data, untrusted content and external communication.
Use three layers. A provider filter such as Azure AI Content Safety or Amazon Bedrock Guardrails covers baseline safety. Llama Guard 4 or NVIDIA NeMo Guardrails handles custom input, output and retrieval checks.
Deterministic policy code governs tool calls, since it is exact and auditable. Put them behind an AI gateway and measure each layer with red team attack success rates.
You get a threat model crosswalk across OWASP LLM 2025, OWASP Agentic 2026 and NIST AI 100-2, a five-step threat modeling method, layered injection defenses, an agent permission decision table, a guardrail comparison, a red teaming program, an incident response playbook, a 13-item pre-launch checklist and an eight-domain maturity scorecard mapped to NIST AI RMF, SOC 2 and ISO/IEC 42001.
It reaches as far as its sources: IBM breach surveys, Gartner predictions, public OWASP, MITRE and NIST frameworks, one NeurIPS jailbreak study and disclosed incidents such as EchoLeak and the Replit database deletion.
The Gartner figures are forecasts, not measurements. None of it assesses your own deployment, which is what the maturity scorecard and a threat model of your data flow are for.
Drop your details and we'll send The AI Security Handbook for Production LLM Apps straight to your inbox - no spam, unsubscribe anytime.
Bring the data flow for one LLM or agent app you run or plan to ship, along with the tools and data it can touch. We will work through a threat model mapped to the OWASP LLM and agentic Top 10 lists and MITRE ATLAS, and mark where authorization, retrieval scoping and approval gates need to sit. SECTION 7 - FAQ - 5 to 8 questions
Book a threat modeling session