Advanced RAG: Hybrid Search, Reranking & Query Transformation

You built a RAG pipeline. Embed the docs, embed the question, cosine-sort, stuff the top chunks into the prompt. It demoed beautifully. Then real users asked real questions and it started confidently citing the wrong paragraph.

That is not a bug you fix by buying a bigger context window. Naive RAG breaks for structural reasons: attention is a finite budget that dilutes as the context grows, and a semantically similar-but-wrong chunk is harder to reject than random noise. The fix is not more retrieval — it is better retrieval, then a precision pass on top.

This walks through the machinery that separates a toy RAG demo from a production one: hybrid search fused by Reciprocal Rank Fusion, a two-stage retrieve-then-rerank pipeline, query transformation, and the frontier past vectors entirely. Every mechanism here is one you can step through live in the interactive deep dive on YeetCode, where the scores are computed from an editable corpus rather than hardcoded.

Why bigger context windows didn't save RAG#

Two failure modes quietly wreck accuracy on production data, and neither goes away when the window grows from 8K to 200K tokens.

Context rot. A transformer does not treat the 10,000th token as reliably as the 100th. Model per-token attention mass as a distribution that sums to 1 across all N positions — the literal meaning of "finite budget." A useful kernel is primacy plus recency:

This exponential-decay-from-both-ends shape reproduces the empirical U-curve from Liu et al.'s Lost in the Middle: accuracy is highest at the start and end of a long context, worst in the middle. Because the weights are normalized, every position's share scales like 1/N. A fact is "attended" only if its per-token weight clears an absolute floor θ, and growing N drops the middle below θ first, then the ends — so a multi-hop question needing two buried facts to connect fails the moment either falls out of budget, no matter how much window you paid for.

Distractors. Retrieval doesn't just have to find the answer; it has to reject look-alikes. Take five candidates with identical surface structure — "Yuki lives beside the Semper Opera," "Mara lives beside the Sydney Opera," and so on — and ask which character lives beside an opera house in a German city. Under a semantic embedding they all map to the same concept vector, so cosine scores them an exact tie: the retriever cannot tell the needle from the four distractors, and top-1 falls to whichever was listed first. On-topic documents are harder than random text precisely because they crowd the answer at the same similarity.

Hybrid search: keyword precision meets semantic recall#

Two retrievers fail in opposite directions, so production stacks run both.

BM25 is Okapi lexical scoring — literal token match, weighted by how rare each term is (IDF). Its formula rewards documents that contain the query's unusual tokens:

With k1 = 1.5 and b = 0.75, a rare literal like a status code 404 carries huge IDF and shoots the doc containing it to the top — even when that doc is topically off. That exactness is BM25's strength and its blind spot.

Vector search embeds query and doc, then ranks by cosine. A trained encoder maps synonyms to nearby vectors, so "members cannot sign in" retrieves a doc about "authentication failing" with no shared literal words. Its blind spot mirrors BM25's: a rare code the encoder never learned sits far from every doc.

You cannot merge their scores directly — BM25 outputs unbounded sums, cosine outputs [-1, 1]. Reciprocal Rank Fusion sidesteps calibration by fusing rankings, not scores:

A doc ranked decently by both retrievers beats one ranked #1 by a single lane, so RRF rewards consensus and dampens each lane's failure. K controls sharpness: small K makes the top rank dominate; large K flattens contributions toward equal.

Walkthrough: fusing two rankings for "404 error on login"#

Six support docs, the query 404 error on login, K = 60. First, run each retriever independently.

BM25 (literal match) ranks:

BM25's top pick is a release note — it matches only because it happens to contain the string "404." A false friend.

Vector (concept match) ranks:

Vector correctly surfaces the actual login-failure doc, which shares no literal token with the query. Now fuse the two rankings (rank is 0-based, so the top doc is rank 0):

The false friend that BM25 loved (#1) drops to #3 after fusion. The real login-failure doc — buried at BM25 rank 2 because it uses zero of the query's literal tokens — climbs to #1 because it earned a strong position in both lanes it could. That is the whole point of RRF: neither lane alone is right, and consensus reconciles them.

Reranking: retrieve cheap, re-sort expensive#

Hybrid search gets the answer into a candidate pool. Getting it to the top is a second problem — the retriever that scales to millions of docs is also the least precise one.

A bi-encoder pre-computes one averaged vector per document, independent of the query — cheap, cacheable, high recall. But averaging blurs a long doc: a runbook whose answer sentence sits amid deployment and on-call filler gets a diluted pooled vector, so it can rank below a shorter, more topically-focused doc. In the deep dive's example, the true answer to "why does an expired token return 419" lands at rank #2 of 5 under bi-encoder cosine, edged out by a short doc that just explains status codes.

The fix is a cross-encoder second pass over only the top-N recall set. Instead of comparing two pre-pooled vectors, it reads query and doc together at token level. A faithful in-browser proxy is ColBERT-style late-interaction MaxSim:

Each query token keeps its single best per-token match anywhere in the doc, then sum. A doc whose individual tokens each align with the query scores high even if its averaged vector was mediocre — so the buried runbook climbs from #2 to #1. You only pay this expensive step N times, not N-million:

The top-N slider is the recall/cost dial: bigger N is safer against dropping the answer, but a tight pool that excludes it at stage 1 is unrecoverable — the reranker can only re-sort what recall handed it.

Query transformation: writing a better query with HyDE#

Sometimes the query itself is the problem. A terse user question — "419 on submit?" — shares little vocabulary with the verbose document that answers it. HyDE (Hypothetical Document Embeddings) fixes this by asking the LLM to hallucinate an ideal answer first, then embedding that hypothetical document instead of the raw question. The fabricated answer, even if factually rough, lives in the same neighborhood as real answers, so its vector retrieves far better than the sparse question did. You are transforming the query into the shape of its target before you ever search.

Beyond vectors: PageIndex and the RAG spectrum#

Chunking is the original sin of naive RAG. Cutting a document on a fixed token count severs relationships that span the boundary. Ask for a launch date and a small chunk size can drop "launch window" into one chunk and "March 15 2024" into the next — the top-ranked chunk is confident, on-topic, and missing the answer. Bigger chunks fix that case and blunt precision everywhere else.

The retrieval spectrum trends toward giving the model control over its own context:

PageIndex keeps the document's structure and lets the LLM navigate it like a human flipping to a chapter: at each level it scores every child heading by IDF-weighted relevance and descends the best match, reaching an intact leaf node where the cue and its date were never separated. On FinanceBench, this approach lifted accuracy from a 54% vector baseline to 98.7% (PageIndex on FinanceBench, Vectify AI, 2024) — cited, not computed in your browser.

What to remember#

• Naive RAG breaks structurally: context rot dilutes attention as 1/N, and distractors tie with the answer under a single similarity metric.
• Run BM25 and vectors. Merge them with RRF (1/(K+rank), K=60) — it needs no shared scale, just orderings, and rewards docs both lanes agree on.
• Retrieve cheaply for recall over everything; rerank expensively for precision over the top-N. A cross-encoder recovers answers a pooled bi-encoder vector buries.
• HyDE transforms a sparse query into the shape of its answer before searching.
• When chunking keeps severing answers, stop chunking — navigate the document's structure instead.

Keep going#

• Recursive Language Models: Context as a Variable — the RLM endpoint of the spectrum, where the model treats its context as an environment to explore.
• How LLMs Actually Work: The Next-Token Loop — the attention mechanics that make context rot a real constraint, not a metaphor.
• How to Read AI Research Papers Without Drowning — for chasing down the ColBERT, RRF, and Lost-in-the-Middle papers behind these ideas.
• From Hand-Written Rules to LLMs: How AI Learned to Learn — the longer arc that RAG's retrieval spectrum echoes.

Then open the interactive deep dive to edit the corpus, drag the RRF constant, resize chunks, and watch every rank recompute live.

FAQ#

What is the difference between hybrid search and reranking?#

They solve different halves of the problem. Hybrid search is recall in the first pass — running BM25 and vector retrieval together and fusing their rankings so the answer lands somewhere in a candidate pool, catching both exact-term and synonym matches. Reranking is a second pass for precision — re-scoring that small pool with an expensive cross-encoder that reads query and doc together, to lift the true answer toward the top. You typically use both: hybrid search builds the recall set, reranking orders it.

Why use Reciprocal Rank Fusion instead of just averaging scores?#

Because BM25 and cosine scores live on incompatible scales — BM25 is an unbounded sum, cosine is bounded in [-1, 1] — so averaging them is meaningless without per-query calibration. RRF fuses the rankings instead: each system contributes 1/(K + rank), which depends only on ordering. That makes it calibration-free and robust, and it rewards documents both retrievers rank well over ones a single lane spikes. The canonical K is 60; smaller values sharpen the top rank's dominance.

What is context rot and does a bigger context window fix it?#

Context rot is the degradation of retrieval reliability as a document sits deeper in a long context. Attention behaves like a fixed budget normalized to sum to 1, so each token's share falls roughly as 1/N as the context grows, and accuracy is empirically U-shaped — best at the start and end, worst in the middle. A bigger window makes it worse: more tokens split the same budget, so a fact retrievable at 1K tokens can fall below the attention floor at 8K. The remedy is retrieving fewer, better chunks, not stuffing more in.

What is a cross-encoder and why is it slow?#

A cross-encoder scores a query-document pair by feeding both through the model together, so every query token can attend to every doc token — that joint interaction is what makes it precise. A bi-encoder, by contrast, encodes each doc once into a pooled vector ahead of time and just does a cosine at query time, which is fast and cacheable but blurs long documents. The cross-encoder's cost is that it must run per pair at query time and cannot be precomputed — exactly why you only apply it to the top-N candidates a cheap retriever already narrowed down.

Is RAG dead now that context windows are huge?#

Naive RAG is dying; retrieval is not. Long-context models still suffer context rot and distractor confusion, so dumping an entire corpus into the prompt underperforms targeted retrieval and costs far more tokens. The trajectory is toward smarter retrieval — hybrid search, reranking, and structure-aware methods like PageIndex that navigate a document tree instead of chunking it — with FinanceBench showing structure-based retrieval jumping from 54% to 98.7% over a vector baseline. Retrieval is evolving, not disappearing.