Most enterprises now ground language models in their own data with retrieval augmented generation, yet many projects stall between demo and production. Menlo Ventures found RAG adoption among US enterprise AI deployments rose from 31% to 51% in a single year, while Gartner predicted at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025.
This paper is a reference architecture for the serving side: the request path, latency and cost, evaluation, groundedness, tenant isolation, observability and failure handling, mapped onto AWS, Azure and Google Cloud. Its numbers come from published research and vendor benchmarks, not from a survey of production systems, and vendor figures should be validated on your own data. Ingestion, chunking and vector store choice sit in companion papers.
Managed services on AWS, Azure and Google Cloud shorten the time to a first answer, but teams usually still own the evaluation harness, the tracing and the tenant controls. Those three decide whether a quality drop, a bad answer or a data leak can be traced and fixed. Each one is checkable through a single named artifact.
Evaluate retrieval and generation separately. Keep a golden set of real questions with expected sources, including unanswerable and adversarial cases, and version it with the index snapshot, prompts and model IDs. Score context recall, faithfulness and answer relevance, calibrate any LLM judge against human labels, and fail the build on regression. Run retrieval metrics first, since a drop there explains most answer regressions. The artifact is a golden set wired into CI with blocking thresholds.
Emit one OpenTelemetry trace per request with child spans for rewrite, retrieval, rerank, generation and verification. The GenAI semantic conventions are still at Development status, so pin versions, but Langfuse and Arize Phoenix both accept OTel traces today. Record passage IDs, index version, token counts, cost and groundedness score on every request, because a bad answer that cannot be reproduced cannot be fixed. The artifact is a trace linking each answer to its passages, cost and user feedback.
Pick an isolation model per risk tier: an index per tenant, a shared index with a mandatory tenant filter, or a hybrid of the two. Enforce the filter in a server-side retrieval service, not in the client, and extend it to caches, traces and evaluation datasets. A shared semantic cache without tenant scoping will serve one user's answer to another. The artifact is a per-tenant red-team test suite for cross-tenant leakage and indirect injection.
Spend latency where it buys accuracy, mostly reranking and generation. Track cost per query by stage, tenant and route, since input tokens from retrieved context usually dominate the bill.
On Microsoft's benchmarks the combination raised NDCG@3 from 51.2 to 72.8 over hybrid search alone. Record the lift on your own golden set before paying for a bigger generator.
Treat a citation to a passage the model never saw as a defect. Run fast groundedness checks inline and deeper checks asynchronously, and return a refusal with sources when evidence is weak.
Rate retrieval, grounding, evaluation, observability, security, tenant isolation, cost control and resilience from 0 to 3. Launch only when no dimension scores 0 and security and evaluation score at least 2.
Retrieval augmented generation is a method where a language model retrieves relevant passages from an external index at query time and uses them to write its answer. Lewis et al. introduced the term in 2020.
It lets you update knowledge by reindexing rather than retraining, cite sources and enforce access rules, which is why it has become the default way enterprises ground models in their own data.
A production RAG architecture has an offline ingestion path and an online request path with eight stages: guard and authenticate, rewrite the query, route, retrieve, rerank, assemble context, generate, and verify and cite.
Around that path sit caching, OpenTelemetry tracing, CI evaluation, online monitoring, tenant isolation and cost controls. This paper covers the online path, and ingestion is covered in a companion paper.
Route simple requests away from retrieval or to smaller models, use a dedicated reranker so fewer and better passages reach the generator, and cache stable prompt prefixes. Anthropic measured a 90% cost cut and time to first token falling from 11.5 seconds to 2.4 seconds on a cached 100,000-token prompt.
Add tenant-scoped retrieval and response caches, and track cost per query by stage, tenant and route.
You can reduce hallucinations but not eliminate them. Stanford researchers found commercial legal RAG tools gave incorrect information in more than 17% of benchmark queries. Improve retrieval first, instruct the model to answer only from supplied context with citations, validate those citations in code, and run a groundedness check such as Bedrock's contextual grounding check or Azure groundedness detection. When evidence is weak, return a refusal with sources.
Use RAG for large, changing or permissioned knowledge that needs citations, fine-tuning to change format, tone or task behavior, and long context for a small document set in one session. Li et al. found a router that tries RAG first and escalates to long context only when needed cut cost by 39% to 65% at comparable quality. Many production systems combine RAG with a lightly tuned model.
You get a reference architecture for serving RAG in production. It covers the eight-stage request path, a latency budget by stage, a cost-per-query model, CI and online evaluation, groundedness and tenant isolation controls, OpenTelemetry tracing, a failure-mode table, reference designs for AWS, Azure and Google Cloud, a 12-step readiness checklist and an eight-dimension scorecard.
It reaches as far as published research and vendor documentation, and no further. Several figures, including Microsoft's ranking results and Anthropic's caching and retrieval numbers, are vendor benchmarks that should be validated on your own data.
The hallucination rate comes from legal research tools. The scorecard cut-offs are a planning heuristic rather than an industry standard, and the paper reports no measured results from Logiciel engagements.
Drop your details and we'll send Production RAG Reference Architecture straight to your inbox - no spam, unsubscribe anytime.
Bring your current RAG pipeline and the traces or evaluation results you already have. We will work through the readiness scorecard with your team and map which stage, from retrieval to grounding to cost, is carrying the weakness. SECTION 7 - FAQ - 5 to 8 questions
Book an architecture review