RAG: retrieve evidence before generation

Section 01

Retrieving evidence before generation

Retrieval-Augmented Generation (RAG) is the practice of finding relevant external evidence and adding it to the model's context before generation. The model then generates a response that can reference, quote, and be constrained by that evidence.

RAG does not write retrieved facts into the model's weights. It adds evidence to the request context at inference time. This distinction matters: RAG gives the model fresh, specific information without retraining, but the model may still ignore, misinterpret, or misrepresent what was retrieved.

Section 02

The archive drawer and the workbench

Think of RAG as a craftsman who, before answering a question, walks to the archive, opens specific drawers, selects a few reference pages, brings them to the workbench, and then crafts a response with those pages visible.

The quality of the response depends on three things: whether the right pages were selected (retrieval quality), whether the pages actually fit on the workbench (context budgeting), and whether the craftsman actually reads and faithfully represents them (generation faithfulness).

Where the analogy breaks: a human craftsman knows when they do not have the right reference. A model does not. It generates a confident response regardless of whether the retrieved evidence is relevant, complete, or even present. The model has no concept of 'I don't have enough information' unless you explicitly design for it.

Section 03

The RAG pipeline

A production RAG system is a pipeline with distinct stages, each independently measurable:

1. INGESTION
   Parse documents (PDF, HTML, text, code)
   Chunk into units (sections, paragraphs, fixed-size windows)

2. EMBEDDING
   Embed each chunk with a versioned embedding model
   Store vectors + metadata in a vector index

3. QUERY
   Embed the user query with the same model
   Retrieve top-k candidates (dense, lexical, or hybrid)
   Apply authorization filters (ACL checks)
   Optionally rerank with a cross-encoder

4. CONTEXT ASSEMBLY
   Pack retrieved chunks into the context window
   Add system instructions and conversation history
   Reserve output space

5. GENERATION
   Model generates response grounded in context
   Include citation instructions in the prompt

6. VERIFICATION
   Check that cited passages exist and support claims
   Measure retrieval quality and answer faithfulness separately

Section 04

Retrieval quality vs answer quality

The most common RAG mistake is measuring only the final answer. If the answer is wrong, you do not know whether retrieval failed (the right evidence was not retrieved) or generation failed (the evidence was there but the model ignored or misrepresented it).

Measure retrieval quality separately: of the retrieved chunks, how many were relevant? Were the relevant chunks ranked highly enough to make it into the context? This requires a labeled evaluation set — queries with known relevant documents.

Measure answer faithfulness separately: does the response accurately reflect the retrieved evidence? Does it make claims not supported by any retrieved chunk? Does it cite real passages? Model-based graders can help, but require calibration.

Section 05

Chunking, authorization, and reranking

Chunking strategy affects retrieval quality. Chunks that are too large retrieve irrelevant context alongside the relevant passage. Chunks that are too small lose context that disambiguates meaning. Common strategies: fixed-size windows with overlap, section-based chunking, sentence-based chunking, or recursive splitting that respects document structure.

Authorization is not optional. The retrieval pipeline must check whether the user is authorized to access each retrieved document before including it in context. A model that cites a document the user should not see is a security failure, not just a quality issue.

Reranking with a cross-encoder improves precision. The initial retrieval (dense or hybrid) is fast but approximate. A cross-encoder re-scores the top-k candidates by jointly encoding the query and each candidate, producing a more accurate relevance score. This adds latency but significantly improves the quality of evidence that reaches the model.

Section 06

Where RAG breaks

Retrieval can miss relevant evidence (recall failure), retrieve irrelevant evidence (precision failure), or retrieve evidence the user is not authorized to see (security failure).

Context assembly can overcrowd the window, pushing out system instructions or the user's actual question. It can also include conflicting evidence from different sources, leaving the model to guess which is authoritative.

Generation can ignore retrieved evidence entirely (the model generates from its parametric knowledge instead). It can hallucinate citations — citing a real source for a claim that source does not make. It can cherry-pick: selecting one detail from evidence while ignoring a contradicting detail in the same document.

Section 07

Unscored lab: trace a RAG pipeline

Design a RAG system for a small knowledge base (10-20 documents).

1. Choose documents. Define your chunking strategy. How big are chunks? How much overlap?
2. Choose an embedding model. Record its version.
3. Write 5 test queries. For each, manually identify which document chunk(s) contain the answer.
4. Run retrieval for each query. Did the relevant chunk make the top 5? Top 3? Top 1?
5. For one query where retrieval failed: why? Was the chunk too large? The embedding model wrong? The query phrased differently from the document?
6. Generate answers with retrieved context. Do the answers cite real passages? Do the cited passages support the claims?
7. Design a verification step: what would you check programmatically?