RAG (Retrieval-Augmented Generation).

Key Takeaways

• RAG retrieves relevant content from a knowledge source and feeds it into an LLM so the model generates grounded, source-backed answers.
• The pattern solves two key problems: integrating knowledge the model lacks and reducing hallucination by grounding outputs in real sources.
• Quality depends heavily on retrieval, not just the model; better chunking, embeddings, hybrid search, and reranking often matter more than which LLM you use.
• The standard stack uses a vector database, an embedding model, optional reranking, and an LLM that supports tool use or large context windows.
• Common failure modes include retrieving wrong chunks, retrieving correct chunks the model ignores, and indexing data with poor chunking that destroys semantic units.
• RAG dominates enterprise AI use cases like search, support, and document Q&A; it is rarely the right answer for tasks where current model knowledge is sufficient.

How RAG Actually Works

The pipeline has four main stages. First, ingest source content into a knowledge store. Documents get chunked into smaller pieces, each chunk gets embedded into a vector, and the vectors are stored in a database optimized for similarity search.

Second, embed the user's query into the same vector space. Most teams use the same embedding model for both indexing and querying to keep results consistent.

Third, search the vector database for chunks similar to the query. The result is typically the top 5 to 20 chunks ranked by cosine similarity. Hybrid search adds keyword matching alongside semantic search to catch cases where exact terms matter.

Fourth, feed the retrieved chunks plus the query to the LLM with a prompt that instructs the model to answer using only the provided context. The model generates the response, often including citations.

Production RAG systems add layers. Reranking takes the top 50 retrieved chunks and uses a more expensive model to score which 5 to 10 are actually relevant. Query rewriting transforms the user's phrasing into something more retrievable. Chunk preprocessing applies summarization or expansion. Post-processing validates that citations point to real sources.

Why Retrieval Quality Drives RAG Performance

The model can only answer well if it gets good context. Retrieve the wrong chunks and you get confidently wrong answers that look authoritative. Most RAG quality issues trace back to retrieval problems rather than model problems.

Chunking strategy matters more than people expect. Splitting a document at arbitrary character counts can break semantic units mid-sentence or mid-thought. Better strategies preserve paragraphs, sections, or other natural boundaries. Some teams use hierarchical chunking that retrieves the small chunk and includes its parent section as additional context.

Embedding quality affects which chunks rank as similar. Domain-specific embeddings (or fine-tuned ones) often outperform generic models for specialized content. Cohere, Voyage, OpenAI, and Anthropic all offer strong embedding APIs.

Hybrid search combining keyword and semantic search catches cases where pure semantic search misses. Acronyms, names, code identifiers, and specific terms benefit from keyword matching alongside semantic similarity.

Reranking dramatically improves precision when the initial retrieval set is large. A reranker model takes the top 50 results and reorders them based on relevance to the query. The cost is a single additional model call per query for substantial quality gains.

Common RAG Architectures

The simplest pattern: a single vector database, a generic embedding model, basic top-k retrieval, and a prompt template that injects the chunks. Works for many use cases and ships in days.

The mid-complexity pattern adds reranking, hybrid search, and metadata filtering. The user's query is rewritten to match the index style. Results are reranked. Metadata filters scope the search to relevant document subsets. This handles harder retrieval problems and broader corpus.

The advanced pattern uses multiple specialized indexes (one per document type or domain), routing queries to the right index, multi-step retrieval (retrieve, then retrieve again based on what was found), and agentic retrieval where the model decides what to search for next. This handles the hardest cases but adds significant complexity.

Most teams start simple and add layers as they hit real quality limits. Premature complexity usually hurts more than it helps.

When to Use RAG (and When Not To)

RAG shines when answers need to come from a specific knowledge source: product documentation, internal wikis, customer records, research papers, legal contracts. It also shines when information needs to be current beyond the model's training cutoff: today's prices, recent events, latest procedures.

RAG is unnecessary when the model already knows what you need. General questions about programming, biology, or history rarely benefit from retrieval; the model has learned enough during training. Adding retrieval wastes tokens and adds latency without improving quality.

RAG struggles with synthesis across many documents. If the answer requires reasoning across hundreds of chunks, retrieval cannot pull all of them into context. Some workflows are better served by document-level summarization first, then RAG over summaries.

RAG also struggles when retrieval cannot find the answer. If your knowledge source genuinely does not contain the answer, the model will either hallucinate or correctly say "I do not know." Both outcomes depend on prompt design and validation.

Best Practices

• Invest in retrieval quality before optimizing the model; better chunking, embeddings, and reranking usually produce larger gains than swapping models.
• Use hybrid search combining keyword and semantic where exact terms matter; pure semantic search often misses precise lookups.
• Apply reranking when retrieval recall is high but precision is low; a reranker on top-50 dramatically improves the relevance of top-5.
• Design prompts to ground the model in retrieved content and instruct it to refuse when context is insufficient; prompt design is part of retrieval quality.
• Build evaluation that scores both retrieval (did we get the right chunks) and generation (did the model answer correctly given the chunks); these are different failure modes that need separate measurement.

Common Misconceptions

• A bigger model fixes RAG quality; in practice, retrieval quality usually limits the system more than model quality does.
• RAG eliminates hallucination; it reduces hallucination significantly but does not eliminate it, especially when retrieval misses or the model strays from provided context.
• Vector search alone is sufficient; hybrid search and reranking improve precision significantly for many production use cases.
• More retrieved chunks always help; beyond a point, additional chunks dilute focus and increase cost without improving quality.
• Standard RAG works for any document corpus; specialized content (code, legal, medical) often needs domain-specific embeddings and chunking strategies.