How Retrieval-Augmented Generation (RAG) Works

An LLM is confidently wrong about your data. It learned from a snapshot of the public web, so it has never seen your company's docs, last week's release notes, or the support ticket a customer is asking about. Ask it a question about any of that and it either admits it does not know or, worse, invents a plausible answer. Retrieval-augmented generation is how you fix that without retraining the model: fetch the relevant text at query time and hand it to the model inside the prompt. The model stops guessing and starts answering from sources you control.

Why not fine-tune or just use a bigger context window?

Two alternatives come up first, and knowing why they fall short is half of understanding RAG.

Fine-tuning adjusts the model's weights on your examples. It is good at teaching a style, a format, or a narrow skill, but it is a poor way to load facts. It is expensive to run, it goes stale the moment your data changes, and it gives you no way to cite where an answer came from. You do not fine-tune every time a document is edited.

Dumping your entire knowledge base into a long context window is the other instinct. But cost and latency scale with every token you send, models degrade when the useful fact is buried in the middle of a huge prompt, and you still have to decide what to include. RAG is the disciplined version of that instinct: select the few passages that actually matter, then generate.

Two pipelines: indexing and answering

RAG is really two pipelines. One runs offline and prepares your documents for search. The other runs on every question. They meet at the vector database.

Indexing: turn documents into searchable vectors

This pipeline runs once per document, and again whenever a document changes. It splits your content into passages, converts each into a vector, and stores it.

A chunk is a bounded passage of text, a few hundred words, small enough to represent one idea. An embedding is a vector, a list of numbers produced by an embedding model, positioned so that text with similar meaning lands close together in vector space. The vector database (pgvector, FAISS, Pinecone, and others) stores each vector alongside the original text and any metadata, and it is built to find nearest neighbors fast.

# Offline: turn your documents into searchable vectors, once per document.
for doc in load_documents():
    for chunk in split(doc, size=800, overlap=100):
        vector = embed(chunk.text)           # same model you will use at query time
        store.upsert(
            id=chunk.id,
            vector=vector,
            text=chunk.text,                  # keep the original text to feed the LLM later
            metadata={"source": doc.source},
        )

Answering: retrieve, then generate

Every question runs the second pipeline. Embed the question with the same model, search the store for the nearest chunks, and put those chunks into the prompt as context. The model answers from what you retrieved.

The retrieval step returns the top-k chunks, the k nearest vectors to the question, usually ranked by cosine similarity. Those chunks, not the model's memory, are what the answer is built from.

def answer(question):
    q_vector = embed(question)                        # same embedding model as indexing
    chunks = store.search(q_vector, top_k=5)          # nearest neighbors by cosine similarity
    context = "\n\n".join(c.text for c in chunks)
    prompt = (
        "Answer using only the context below. "
        "If the answer is not there, say you do not know. Cite sources.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return llm.generate(prompt), [c.metadata["source"] for c in chunks]

That instruction to answer only from the context, admit ignorance, and cite sources is what turns a fluent guesser into a grounded one. The context does the knowing; the model does the wording.

The decisions that make or break RAG

A basic pipeline is a weekend project. A RAG system that gives correct, trustworthy answers lives or dies on these choices.

Chunking is the sleeper variable

Chunk too large and retrieval drags in noise that dilutes the real signal and wastes context. Chunk too small and each piece loses the surrounding context that gave it meaning, so a passage retrieved alone no longer answers anything. Add a little overlap between neighboring chunks so a sentence split across a boundary is not lost, and split on natural structure, headings and paragraphs, rather than a blind character count. More retrieval failures trace back to bad chunking than to any other single cause.

Retrieval quality: hybrid search and reranking

Pure vector search matches meaning, which is exactly why it fumbles exact terms: product codes, error strings, rare names, an order id. Keyword search (BM25) catches those literal matches. Run both and merge the results, and you have hybrid search. Then pass the merged candidates through a reranker, a model that scores each chunk against the question directly and reorders them, so the strongest few rise to the top of the prompt.

The model can only cite what it receives. If the right chunk never makes the top-k, no amount of prompt engineering saves the answer.

Grounding and citations

Tell the model to answer strictly from the retrieved context, to say it does not know when the context is silent, and to cite the chunk each claim came from. Return those sources to the user. A cited answer is one a reader can verify, and an "I do not know" is far cheaper than a confident fabrication.

Evaluate retrieval and generation separately

RAG has two failure points, and lumping them together hides which one is broken. Retrieval either surfaced the right chunk in the top-k or it did not, which you measure with recall. Generation either stayed faithful to the retrieved context or it drifted, which you measure with a faithfulness or groundedness check. A good answer sitting on top of bad retrieval is luck, not a working system, so track both numbers.

Keep the index fresh

Re-embed and re-index when documents change. A stale index answers from yesterday's truth, which is the precise failure you adopted RAG to escape. Freshness is the whole point, so treat reindexing as part of the pipeline, not an afterthought.

When RAG is the wrong tool

RAG earns its complexity on a large, changing, private body of knowledge where answers must be grounded and cited. Outside that, simpler wins. If your reference material is small and stable, just paste it into the prompt. If you need a new tone, format, or skill rather than new facts, fine-tune. And if a question needs multi-hop reasoning across many documents, plain top-k retrieval will struggle, which is where agentic and graph-based retrieval pick up. Reach for RAG when the knowledge is too big to prompt, too fresh to train, and too important to guess.

Reading the pipeline is not the same as running one. To build RAG end to end, chunking, embeddings, a vector store, hybrid retrieval, and evaluation on real documents, work through the AI Engineering course.