Logiciel Solutions Contact Us
Success Stories Tech News Investors Contact Us
whitepaper
8 min read

Production RAG Reference Architecture.

Most enterprises now ground language models in their own data with retrieval augmented generation, yet many projects stall between demo and production. Menlo Ventures found RAG adoption among US enterprise AI deployments rose from 31% to 51% in a single year, while Gartner predicted at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025.

This paper is a reference architecture for the serving side: the request path, latency and cost, evaluation, groundedness, tenant isolation, observability and failure handling, mapped onto AWS, Azure and Google Cloud. Its numbers come from published research and vendor benchmarks, not from a survey of production systems, and vendor figures should be validated on your own data. Ingestion, chunking and vector store choice sit in companion papers.

In depth

A Demo Calls A Model, A Production System Runs Eight Stages.

01

Production RAG is a multi-stage service, not a single model call.

Retrieval augmented generation answers using passages fetched at query time from an external index rather than only from trained weights, a design Lewis et al. formalized in 2020 when they treated the index as memory that could be swapped without retraining. In production that becomes eight stages: guard and authenticate, rewrite the query, route, retrieve, rerank, assemble context, generate, then verify and cite. Each stage is a separate span in the traces, a separate line in the cost model and a separate place to fail. A production system also adds what research setups skip, including per-user access control, freshness, provenance, audit logs, cost limits and on-call ownership. A notebook that calls a vector store and a model is still a prototype.

In shortA notebook that calls a vector store and a model is…
02

Reranking moves quality more than a new embedding model.

On Microsoft's internal datasets, hybrid search with text-embedding-3-large scored 51.2 NDCG@3, rising to 72.8 once query rewriting and the new semantic ranker were added. The same stack lifted an older ada-002 index to 71.7, close to the newer embedding model. Anthropic's contextual retrieval work points the same way: contextual embeddings plus contextual BM25 cut top-20 retrieval failures by 49%, and adding a reranker raised the reduction to 67%. Both are vendor benchmarks and should be checked on your own data. Order still matters after retrieval, since Liu et al. found accuracy is highest when the relevant passage sits at the start or end of the prompt. A good reranker can postpone a costly re-embedding project.

In shortA good reranker can postpone a costly re-embedding p…
03

Prompt caching is the largest single cost lever for long, repeated context.

Anthropic measured a 90% cost cut and time to first token falling from 11.5 seconds to 2.4 seconds on a cached 100,000-token prompt, with cost cuts of 53% to 90% across workloads. Cost per query is a sum of stage costs: rewrite tokens, query embedding, search, reranker calls, generator input and output tokens, groundedness checks and trace storage. Input tokens from retrieved context usually dominate, so top-k and passage length are cost settings as much as quality settings. Long context does not remove the need for retrieval either. Li et al. found a router that tried RAG first and escalated to long context only when needed cut cost by 39% for GPT-4o and 65% for Gemini 1.5 Pro at comparable quality. Reranking and caching usually return more quality per dollar than upgrading the generator.

In shortReranking and caching usually return more quality pe…
04

RAG lowers hallucination risk but does not remove it.

Stanford RegLab and HAI researchers found Lexis+ AI and Ask Practical Law AI gave incorrect information in more than 17% of benchmark queries, including misgrounding, where a correct-sounding statement is backed by a source that does not support it. Managed checks now exist on each cloud: the Bedrock Guardrails contextual grounding check, Azure AI Content Safety groundedness detection, and Vertex AI RAG Engine grounding paired with a calibrated judge model. Retrieved text is also an attack surface, since the OWASP 2025 Top 10 for LLM applications lists indirect prompt injection under LLM01 and vector and embedding weaknesses as LLM08. When the grounding check fails, the system should return a safe refusal with its best sources rather than a guess. Groundedness checks belong in the request path.

In shortGroundedness checks belong in the request path
The detail

Three Controls Every Production RAG System Needs Before Launch.

Managed services on AWS, Azure and Google Cloud shorten the time to a first answer, but teams usually still own the evaluation harness, the tracing and the tenant controls. Those three decide whether a quality drop, a bad answer or a data leak can be traced and fixed. Each one is checkable through a single named artifact.

Zone · 01

A versioned golden set in CI

Evaluate retrieval and generation separately. Keep a golden set of real questions with expected sources, including unanswerable and adversarial cases, and version it with the index snapshot, prompts and model IDs. Score context recall, faithfulness and answer relevance, calibrate any LLM judge against human labels, and fail the build on regression. Run retrieval metrics first, since a drop there explains most answer regressions. The artifact is a golden set wired into CI with blocking thresholds.

Zone · 02

One trace per request

Emit one OpenTelemetry trace per request with child spans for rewrite, retrieval, rerank, generation and verification. The GenAI semantic conventions are still at Development status, so pin versions, but Langfuse and Arize Phoenix both accept OTel traces today. Record passage IDs, index version, token counts, cost and groundedness score on every request, because a bad answer that cannot be reproduced cannot be fixed. The artifact is a trace linking each answer to its passages, cost and user feedback.

Zone · 03

Tenant isolation beyond the index

Pick an isolation model per risk tier: an index per tenant, a shared index with a mandatory tenant filter, or a hybrid of the two. Enforce the filter in a server-side retrieval service, not in the client, and extend it to caches, traces and evaluation datasets. A shared semantic cache without tenant scoping will serve one user's answer to another. The artifact is a per-tenant red-team test suite for cross-tenant leakage and indirect injection.

By the numbers

The figures that make it a board-level conversation.

90%
cost cut on a cached 100,000-token prompt, with time to first token down 79%
51.2 to 72.8
NDCG@3 on Microsoft's internal datasets when query rewriting and the new semantic ranker were added to hybrid search
17%+
of benchmark queries answered with incorrect information by commercial legal RAG tools
Inside the report

What you'll take away.

01

Step 1 - Give every pipeline stage its own p95 latency target and cost line

Spend latency where it buys accuracy, mostly reranking and generation. Track cost per query by stage, tenant and route, since input tokens from retrieved context usually dominate the bill.

02

Step 2 - Add a reranker and query rewriting to hybrid search, then measure

On Microsoft's benchmarks the combination raised NDCG@3 from 51.2 to 72.8 over hybrid search alone. Record the lift on your own golden set before paying for a bigger generator.

03

Step 3 - Validate citations in code and refuse when grounding fails

Treat a citation to a passage the model never saw as a defect. Run fast groundedness checks inline and deeper checks asynchronously, and return a refusal with sources when evidence is weak.

04

Step 4 - Score the system on the eight-dimension readiness scorecard

Rate retrieval, grounding, evaluation, observability, security, tenant isolation, cost control and resilience from 0 to 3. Launch only when no dimension scores 0 and security and evaluation score at least 2.

Questions

Frequently asked.

What is retrieval augmented generation?

Retrieval augmented generation is a method where a language model retrieves relevant passages from an external index at query time and uses them to write its answer. Lewis et al. introduced the term in 2020.

It lets you update knowledge by reindexing rather than retraining, cite sources and enforce access rules, which is why it has become the default way enterprises ground models in their own data.

What does a production RAG architecture look like?

A production RAG architecture has an offline ingestion path and an online request path with eight stages: guard and authenticate, rewrite the query, route, retrieve, rerank, assemble context, generate, and verify and cite.

Around that path sit caching, OpenTelemetry tracing, CI evaluation, online monitoring, tenant isolation and cost controls. This paper covers the online path, and ingestion is covered in a companion paper.

How do you reduce RAG latency and cost per query?

Route simple requests away from retrieval or to smaller models, use a dedicated reranker so fewer and better passages reach the generator, and cache stable prompt prefixes. Anthropic measured a 90% cost cut and time to first token falling from 11.5 seconds to 2.4 seconds on a cached 100,000-token prompt.

Add tenant-scoped retrieval and response caches, and track cost per query by stage, tenant and route.

How do you stop RAG from hallucinating?

You can reduce hallucinations but not eliminate them. Stanford researchers found commercial legal RAG tools gave incorrect information in more than 17% of benchmark queries. Improve retrieval first, instruct the model to answer only from supplied context with citations, validate those citations in code, and run a groundedness check such as Bedrock's contextual grounding check or Azure groundedness detection. When evidence is weak, return a refusal with sources.

Should I use fine-tuning, RAG or long context?

Use RAG for large, changing or permissioned knowledge that needs citations, fine-tuning to change format, tone or task behavior, and long context for a small document set in one session. Li et al. found a router that tries RAG first and escalates to long context only when needed cut cost by 39% to 65% at comparable quality. Many production systems combine RAG with a lightly tuned model.

What do I get in the Production RAG Reference Architecture whitepaper?

You get a reference architecture for serving RAG in production. It covers the eight-stage request path, a latency budget by stage, a cost-per-query model, CI and online evaluation, groundedness and tenant isolation controls, OpenTelemetry tracing, a failure-mode table, reference designs for AWS, Azure and Google Cloud, a 12-step readiness checklist and an eight-dimension scorecard.

How far does the evidence in this paper reach?

It reaches as far as published research and vendor documentation, and no further. Several figures, including Microsoft's ranking results and Anthropic's caching and retrieval numbers, are vendor benchmarks that should be validated on your own data.

The hallucination rate comes from legal research tools. The scorecard cut-offs are a planning heuristic rather than an industry standard, and the paper reports no measured results from Logiciel engagements.

Get the whitepaper

Have it emailed to you.

Drop your details and we'll send Production RAG Reference Architecture straight to your inbox - no spam, unsubscribe anytime.

Download whitepaper
Next step

Score your RAG system before your users find its weakest stage.

Bring your current RAG pipeline and the traces or evaluation results you already have. We will work through the readiness scorecard with your team and map which stage, from retrieval to grounding to cost, is carrying the weakness. SECTION 7 - FAQ - 5 to 8 questions

Book an architecture review