RAG Implementation Checklist | 50-Point Production Readiness Check

1. Data & Ingestion

Garbage in, garbage out. Fix the source before you build anything on top.

Every source document has a known owner and a known update cadence Ingestion pulls from the system of record, not a one-off export You strip boilerplate, headers, footers, and navigation before indexing Tables, figures, and code blocks survive parsing instead of turning to mush Scanned and image-heavy files run through OCR with a quality check Each chunk carries metadata: source, title, date, section, access level Duplicate and near-duplicate documents are collapsed, not indexed twice Stale and deleted source content is removed from the index, not just left behind Sensitive data (PII/PHI) is detected and handled per policy before it lands in the index Re-ingestion is incremental, so one new file does not rebuild everything

2. Chunking & Indexing

How you split and store documents determines how well the model can use them.

3. Retrieval

If you retrieve the wrong context, no amount of prompt engineering saves you.

4. Generation & Grounding

The model must answer from the retrieved context — not hallucinate around it.

5. Evaluation

You cannot improve what you do not measure — and users will find the gaps first.

A gold set of real questions with correct answers and source passages exists Retrieval and generation are evaluated separately, not just end to end Faithfulness, context precision, context recall, and answer relevance are tracked as numbers Evaluation runs automatically on every prompt, model, or index change A quality threshold gates deployment in CI/CD; failing scores block the release Regressions are caught by the eval suite, not by users reporting bad answers The gold set is refreshed as the corpus and real questions evolve Human review samples live traffic on a defined cadence Eval results are stored over time so you can see trends, not just a snapshot

6. Production

A demo that answers your three favourite questions is not a system.

Your readiness score

0 / 50

Work through the checklist to see your readiness level.

🔒

Submit your email to unlock your score

Get your RAG production plan

We’ll email a prioritised breakdown of your gaps and a concrete plan to ship retrieval that you can actually trust in production. No pitch.

No spam. We’ll follow up only if it’s relevant.