What Is RAG? Retrieval-Augmented Generation from the Ground Up

Ask a raw LLM "What was ACME Corporation's Q3 2024 revenue?" and you get one of two failures: a confident hallucination, or an honest "I don't have that information." The model was frozen at training time and never saw your internal finance doc, so it either guesses or gives up.

Retrieval-Augmented Generation fixes all three. Instead of asking the model to recall a fact, you look it up first, paste it into the prompt, and ask the model to answer using only what you handed it — a memory bank turned into a reasoning engine over evidence you control.

That reframing — retrieve, then augment, then generate — is the whole idea; everything else is engineering the retrieval step so the right evidence shows up.

Why RAG exists: three failure modes#

A standalone LLM fails in three specific ways, each targeted directly by retrieval.

None of these needs retraining — same model, different input. The interactive companion opens here: pick a failure mode and watch a wrong answer turn grounded.

Retrieve → Augment → Generate#

The loop has three stages, and keeping them separate is what makes RAG debuggable.

1. Retrieve. Take the user's question, search a prepared index, and pull back the top few most relevant chunks.
2. Augment. Splice those chunks into the prompt with an instruction like "Answer using only the passages below; if they don't contain the answer, say so."
3. Generate. The LLM writes the answer, now anchored to real evidence instead of parametric memory.

When a RAG answer is wrong, you can point at the stage that broke. Wrong answer with the right chunk retrieved is a generation problem; wrong answer because the right chunk never showed up is a retrieval problem. That separation is the practical payoff of the architecture.

Chunking: how you slice the document decides everything#

Retrieval doesn't search whole documents — it searches chunks, small passages embedded and indexed independently. How you cut a document into chunks decides whether a question's answer survives intact inside one chunk.

Take this source text the deep dive uses:

Now the test that matters: the answerable-span test. For "How did consulting perform?", the answer is the contiguous span "consulting declined 5%." A chunk is good if that whole span lands inside one chunk, bad if the split cuts through the middle of the fact.

Two strategies handle this differently:

Fixed-size chunking is trivial — slice every size - overlap words — but a cut that lands mid-sentence shreds the fact. Recursive chunking trades that simplicity for boundary awareness. In the interactive version you drag the chunk size and watch the answerable-span score collapse under fixed-size shredding and hold steady under recursive splitting.

Embeddings and vector similarity search#

Once you have chunks, you need to find the ones relevant to a query. Keyword matching is brittle — "revenue" won't match "earnings." Embeddings solve this by turning text into a vector in high-dimensional space where similar meanings sit close together.

The deep dive uses an honest, in-browser embedding you can reason about by hand: a bag-of-words vector over the corpus vocabulary, then L2-normalized to length 1. Identical words become the same unit vector; text with no shared words becomes orthogonal.

Relevance is measured by cosine similarity — the dot product of two unit vectors:

For unit vectors the denominator is 1, so cosine collapses to the plain dot product. Cosine of 1.0 means identical direction; 0 means no shared meaning. To answer a query you embed it, score its cosine against every chunk, sort descending, and keep the top-k nearest neighbors — your retrieved evidence.

Only the top chunk shares two query tokens — ACME and revenue; everything below shares exactly one, so rank comes down to which lone token overlaps and how short the chunk is. Two results surprise here. A people-topic chunk ("added 1200 new employees") outranks two finance chunks purely because it also says ACME. And "Revenue grew 12 percent" — the chunk that reads most on-topic — ties for last among non-zero scores: bag-of-words does no stemming, so grew and growth are different tokens, leaving it sharing only revenue with the query. These are real cosine scores over the live vectors, not a hardcoded result — which is exactly how a naive lexical embedding exposes its own blind spots.

How can retrieval fail on ambiguous chunks?#

Cosine search breaks when a chunk is self-ambiguous. Consider a query and four rival chunks:

The correct chunk (acme) never says "ACME" or "2024." Bare, it shares only revenue and quarter with the query — the same tokens its rivals share — so it does not reliably rank first. The fact is right there; the chunk just lost its who and when when it left the document.

Contextual Retrieval: give each chunk back its who and when#

Anthropic's Contextual Retrieval technique fixes this at ingestion time. Before embedding a chunk, you use an LLM to write a short context blurb situating it in the source document, and you prepend that blurb to the chunk. The acme chunk becomes:

Now the embedded text contains ACME, 2024, quarterly, and figure — the exact disambiguating tokens the query carries. Its cosine to the query rises and it jumps to rank 1, while its still-bare rivals stay put. The interactive block computes that rank change live as you edit the blurb.

Anthropic reported this approach cuts retrieval failures by roughly 67% when combined with reranking. Treat that as published literature, not a number this demo measures — but the mechanism, injecting the missing context so the vector carries it, is exactly what the live cosine jump shows.

The full pipeline and its upgrade path#

Stitch the stages together and a production RAG system looks like this:

The upgrade path is a ladder, and you climb it only when evaluation tells you to:

Each rung changes the retrieved evidence, which changes the grounded answer. The end-to-end block runs a real query through the whole chain, reusing the same chunking and cosine engines from earlier parts, so its retrieval matches the standalone results.

What to remember#

• RAG turns an LLM from a memory bank into a reasoner over evidence you supply: retrieve, augment, generate.
• It fixes three failures — frozen knowledge, private-data blindness, hallucination — with no retraining.
• Chunking quality is decided by the answerable-span test: does the whole fact survive inside one chunk? Recursive beats fixed-size by never cutting a sentence.
• Retrieval is cosine top-k over normalized embeddings; on unit vectors cosine is just a dot product.
• Contextual Retrieval prepends the context a chunk loses when sliced out, and the correct chunk's rank climbs.
• Debug by stage: right chunk plus wrong answer is a generation bug; missing chunk is a retrieval bug.

Keep going#

• Advanced RAG: Hybrid Search, Reranking & Query Transformation — the next rungs on the ladder.
• Recursive Language Models: Context as a Variable — a different answer to the frozen-context problem.
• How to Read AI Research Papers Without Drowning — for reading the Contextual Retrieval paper yourself.
• How LLMs Actually Work: The Next-Token Loop — what the "generate" stage is doing under the hood.

Explore it interactively on YeetCode — open the RAG deep dive, drag the chunk size, type your own queries, and watch the cosine ranks move.

FAQ#

What does RAG stand for and what problem does it solve?#

RAG stands for Retrieval-Augmented Generation. It's an architecture, not a model: you bolt a search step onto an ordinary LLM so it fetches relevant text from a corpus you control and answers only from that. Because the weights never change, you can add RAG to any off-the-shelf model without training, and swap in a better model whenever one ships.

How is chunking different from just splitting text into equal pieces?#

Equal-size splitting is one chunking strategy, not a synonym for chunking. Chunking is the general question of where to cut; fixed-size splitting is the laziest answer, cutting at a word count with no regard for structure. People still reach for it because it's fast and uniform, so indexing cost is easy to budget. Overlap — repeating a few words across each boundary — is the usual band-aid, but it wastes index space and only masks the real fix: cutting on paragraph and sentence boundaries instead of arbitrary offsets.

Why use cosine similarity instead of keyword matching for retrieval?#

Cosine scores meaning, so it can surface a chunk about "earnings" for a query about "revenue" — something exact-string matching never manages. But you often want both. Keyword search (classically BM25) still wins on rare, exact tokens: part numbers, error codes, function names, proper nouns that appear verbatim and carry no useful nearby meaning. A purely semantic index can rank a chunk mentioning error code E-4021 below fuzzier matches because the code is a rare token. That's why mature systems fuse cosine and keyword scores into hybrid search rather than pick one.

What is Contextual Retrieval and why does the 67% figure matter?#

The 67% is a reduction in retrieval failures against a naive cosine baseline, not an absolute accuracy — so it only means something relative to the plain top-k it's meant to improve. It's also a cost trade: writing a context blurb for every chunk means one LLM call per chunk at ingestion, and the technique only became affordable because prompt caching lets you reuse the source document cheaply across those calls. Crucially, all of that cost is paid once, offline — query-time latency is unchanged, since the context is already baked into the stored vector by the time a question arrives.

How do I debug a RAG system that gives wrong answers?#

Log the retrieved chunks for every query first — most RAG bugs stay invisible until you print the top-k. Then two metrics split the failure classes: recall@k (did the chunk holding the answer appear in the retrieved set?) points at retrieval, and a faithfulness check (does the generated claim appear in that retrieved text?) points at generation. Low recall sends you to chunking, embeddings, or a too-small k; high recall with low faithfulness sends you to the grounding prompt. Swapping embedding models when the real bug is a loose prompt is the most common wasted week in RAG work.