Logiciel Solutions Contact Us
Success Stories Tech News Investors Contact Us
whitepaper
8 min read

The AI Security Handbook for Production LLM Apps.

Teams are wiring LLMs and agents into tickets, CRMs, code and payments faster than they are deciding what those agents are allowed to touch. In IBM's 2025 study, 97% of organizations that reported an AI-related breach said they lacked proper AI access controls, and 63% had no AI governance policy.

AI security is the practice of protecting models, prompts, retrieval data, agents and tools from attack and misuse, and this handbook maps it to OWASP, MITRE ATLAS and NIST. The evidence is breach surveys, analyst predictions, public frameworks, one peer-reviewed jailbreak study and disclosed incidents.

It describes where attacks land and which controls limit them, not how secure any particular deployment is, and the Gartner figures are forecasts rather than measurements.

In depth

The Model Cannot Separate Instructions From Data, So Security Lives Outside It.

01

Prompt injection is the top risk because the model reads everything as instructions.

OWASP lists prompt injection as LLM01 in its 2025 Top 10, and NIST AI 100-2 E2025 splits it into direct injection, typed by the user, and indirect injection, hidden in emails, web pages, PDFs or retrieved documents. Indirect injection is the more dangerous of the two because the victim never sees the payload. EchoLeak (CVE-2025-32711), disclosed by Aim Security in June 2025, let a crafted email make Microsoft 365 Copilot exfiltrate internal data with no user click, and Microsoft rated it CVSS 9.3. The attack bypassed existing injection classifiers and link redaction. Delimiters and classifiers reduce the risk without removing it. The working assumption has to be that some injections will succeed.

In shortThe working assumption has to be that some injection…
02

Jailbreak defenses fail on a curve, not on a single attempt.

Model outputs are non-deterministic, so a refusal that holds once can give way on a retry. In Best-of-N jailbreaking, attackers apply random capitalization or shuffling until one variant slips through, and success climbs on a power-law curve as attempts increase. At 10,000 attempts it broke GPT-4o 89% of the time and Claude 3.5 Sonnet 78% of the time. Anthropic's many-shot research found the same pattern with long prompts packed with fake compliant dialogues, where success rises predictably with more examples. One-shot red team tests therefore overstate safety. Rate limits and attempt budgets per user, session and API key matter as much as the classifier itself.

In shortRate limits and attempt budgets per user, session an…
03

Excessive agency turns every other weakness into real damage.

An agent converts text into actions, so a successful injection becomes a transaction, a deletion or an email. Simon Willison's lethal trifecta names the combination to avoid: private data access, untrusted content and an outbound channel in one flow. Gartner calls flows that combine all three a no-go zone and advises agent-specific permissions rather than inherited human roles. In July 2025 a Replit agent deleted a live production database during a code freeze, after which Replit separated development and production data automatically. Gartner also predicts that over half of successful attacks on AI agents will exploit access control weaknesses and prompt injection by 2029. If an agent needs all three trifecta elements, split it with a human or deterministic policy in between.

In shortsplit it with a human or deterministic policy in between
04

Access control is where most AI breaches are lost.

In IBM's 2025 study, 97% of organizations with an AI-related breach lacked proper AI access controls and 63% had no AI governance policy, while high shadow AI usage added $670,000 to the average breach cost. Gartner expects 25% of enterprise GenAI applications to see at least five minor security incidents a year by 2028, up from 9% in 2025, and predicts that 40% of enterprises will demote or decommission autonomous agents by 2027 because governance gaps surface only after production incidents. The fix is ordinary identity engineering applied to agents: a workload identity per agent, on-behalf-of tokens, short-lived credentials and approval gates. Scoping permissions before launch is cheaper than rolling an agent back.

In shortScoping permissions before launch is cheaper than ro…
The detail

Three Questions That Show Whether an LLM App Is Ready for Production.

None of this requires a new security discipline. It requires treating the LLM as an untrusted, highly capable user inside the system and checking that the architecture assumes as much. Three questions test that, each answerable with one named artifact a reviewer can read.

Zone · 01

Who decides

the model or code

Ask where authorization happens when the model proposes an action. The answer should be deterministic code outside the model that checks the requesting user's rights, with human approval for payments, deletions, external email and permission changes. If the model's own judgment is the only gate, one successful injection reaches everything the agent can reach. The artifact is an agent permission table that lists each action type, its default autonomy and its required control.

Zone · 02

What can a hijacked agent reach

Ask what data and tools a fully compromised agent could touch. Retrieval should filter by the caller's identity at query time, indexes should be partitioned per tenant, and no single flow should combine private data, untrusted content and an outbound channel. Anything wider is blast radius the team has accepted without saying so. The artifact is a data flow diagram with every untrusted input and every triggerable action marked.

Zone · 03

How often do the guardrails fail

Ask for the attack success rate per threat category, measured with realistic retry budgets. A team running PyRIT, garak or promptfoo in CI on every prompt, model or tool change can answer in numbers, while a team relying on provider defaults can only say the filter is switched on. Because success climbs with attempts, a one-shot result is not an answer. The artifact is a baseline ASR report with release thresholds attached.

By the numbers

The figures that make it a board-level conversation.

97%
of organizations reporting an AI-related breach said they lacked proper AI access controls, and 63% had no AI governance policy
89%
Best-of-N jailbreak success against GPT-4o at 10,000 attempts, against 78% for Claude 3.5 Sonnet
$4.99 million
global average cost of a data breach, in a year that saw a 56% rise in AI-driven attacks
Inside the report

What you'll take away.

01

Step 1 - Build a threat model mapped to OWASP, ATLAS and NIST IDs

Draw the data flow, mark every point where untrusted text enters the context window, list each triggerable action and assign one preventive control, one detective control and one owner per path.

02

Step 2 - Move authorization out of the model and scope every agent

Give each agent its own workload identity, on-behalf-of tokens and short-lived credentials, allowlist tools and arguments, and require human approval for irreversible, financial or privilege-changing actions.

03

Step 3 - Layer guardrails and measure them with red teaming in CI

Pair a provider filter, a classifier on retrieved content and deterministic policy code for tool calls, then track attack success rate per category with realistic retry budgets and set release thresholds.

04

Step 4 - Score maturity quarterly and reuse one evidence library

Rate eight domains from 1 to 4 on the scorecard, treat a total under 16 as not ready for broad release, and tag controls for NIST AI RMF, SOC 2 and ISO/IEC 42001.

Questions

Frequently asked.

What is AI security?

AI security is the protection of AI systems, including models, prompts, training and retrieval data, agents and tools, against attacks and misuse. For production LLM apps it focuses on prompt injection, data leakage, unsafe outputs, poisoned supply chains and over-privileged agents.

It builds on standard application security and is distinct from using AI to run security operations, which this handbook does not cover.

What is prompt injection and how do you prevent it?

Prompt injection is crafted text, typed by a user or hidden in a document, email or web page, that overrides an LLM's intended instructions. It is LLM01 on OWASP's 2025 list, and no filter fully prevents it.

Reduce the impact by keeping authorization outside the model, scoping data access to the user, filtering inputs, stripping markdown images and hidden links from outputs, and requiring human approval for high-impact actions.

How do attackers jailbreak AI models, and how do you defend against it?

Attackers use role-play framing, many-shot prompts full of fake compliant dialogues, encoding tricks, multi-turn escalation and Best-of-N random variations. Best-of-N reached 89% success on GPT-4o at 10,000 attempts. Defend with layered classifiers, per-user rate limits and retry budgets, continuous red teaming, and an architecture that limits what a jailbroken model can reach, since some attempts will get through.

How do you secure AI agents with least privilege?

Give each agent its own workload identity and use on-behalf-of tokens so it acts with the requesting user's rights. Issue short-lived credentials from a secrets manager, allowlist tools and arguments such as refund caps, and keep agents out of production write paths by default.

Require human approval for payments and deletions, and avoid flows that combine private data, untrusted content and external communication.

Which runtime guardrails should production LLM apps use?

Use three layers. A provider filter such as Azure AI Content Safety or Amazon Bedrock Guardrails covers baseline safety. Llama Guard 4 or NVIDIA NeMo Guardrails handles custom input, output and retrieval checks.

Deterministic policy code governs tool calls, since it is exact and auditable. Put them behind an AI gateway and measure each layer with red team attack success rates.

What do I get in the handbook?

You get a threat model crosswalk across OWASP LLM 2025, OWASP Agentic 2026 and NIST AI 100-2, a five-step threat modeling method, layered injection defenses, an agent permission decision table, a guardrail comparison, a red teaming program, an incident response playbook, a 13-item pre-launch checklist and an eight-domain maturity scorecard mapped to NIST AI RMF, SOC 2 and ISO/IEC 42001.

How far does the evidence in this handbook reach?

It reaches as far as its sources: IBM breach surveys, Gartner predictions, public OWASP, MITRE and NIST frameworks, one NeurIPS jailbreak study and disclosed incidents such as EchoLeak and the Replit database deletion.

The Gartner figures are forecasts, not measurements. None of it assesses your own deployment, which is what the maturity scorecard and a threat model of your data flow are for.

Get the whitepaper

Have it emailed to you.

Drop your details and we'll send The AI Security Handbook for Production LLM Apps straight to your inbox - no spam, unsubscribe anytime.

Download whitepaper
Next step

Find out what a hijacked agent in your stack could reach.

Bring the data flow for one LLM or agent app you run or plan to ship, along with the tools and data it can touch. We will work through a threat model mapped to the OWASP LLM and agentic Top 10 lists and MITRE ATLAS, and mark where authorization, retrieval scoping and approval gates need to sit. SECTION 7 - FAQ - 5 to 8 questions

Book a threat modeling session