RAG: the hard part was never the model

The last post ended on a limit, stated as plainly as I could manage: prompting cannot add knowledge the model doesn't have. No phrasing recovers information that isn't in the weights or the window.

That sentence contains its own escape hatch. Or the window. The weights are frozen — that's what inference means — but the context window is writeable, and nothing anywhere says you have to be the one typing into it.

So: before you send the prompt, go find the text that answers the question, paste it in, and then ask. That's retrieval-augmented generation. That's the entire idea, and if it sounds anticlimactic, hold onto that feeling, because it's the most useful intuition in this post.

Here's the claim I'll defend. The "augmented generation" half of RAG is trivial — it is a prompt with some text pasted into it, and nothing about the model changes. Every genuinely hard problem that remains is a search problem, and most of them were named and studied before the transformer existed. Teams fail at RAG because they staff it as an AI project. It is, in the parts that hurt, an information retrieval project.

What RAG actually is, mechanically

There are two loops, and keeping them separate explains most of what follows.

The indexing loop runs offline, on your documents, whenever they change. You collect the corpus, split each document into chunks, run every chunk through an embedding model, and store the resulting vector alongside the text it came from.

The query loop runs online, once per question. You embed the question with the same model, find the chunks whose vectors sit nearest to it, paste those chunks into a prompt, and send it.

In Postgres with pgvector, the storage side is unremarkable:

create table chunks (
  id        bigserial primary key,
  doc_id    text   not null,
  content   text   not null,
  embedding vector(1536)
);
 
create index on chunks using hnsw (embedding vector_cosine_ops);

And the retrieval is one ordinary query — <=> is cosine distance, so ordering by it ascending gives you nearest-first:

select doc_id, content
from chunks
order by embedding <=> $1
limit 8;

Then the part everyone calls the hard part:

const queryVector = await embed(question);
const chunks = await db.query(NEAREST_CHUNKS, [queryVector]);
 
const answer = await model.complete(`
Answer the question using only the context below.
If the context does not contain the answer, say that.
 
<context>
${chunks.map((c) => c.content).join("\n\n")}
</context>
 
Question: ${question}
`);

That's the whole architecture. Look at what that last call actually is: an ordinary completion request against an ordinary frozen model. No weights were updated. No fine-tuning happened. The <context> tags are a formatting convention the model was post-trained to find legible, not a channel with any special standing — the same point the roles post made about system prompts.

The model cannot tell the difference between a paragraph you pasted by hand and one an HNSW index found forty milliseconds ago. There is no retrieval step inside the model. There is a string, and the string got longer.

Why you'd want this

Four reasons, and they're worth separating because they pull in different directions.

Knowledge the model never saw. Cutoffs handle the time axis, but the bigger gap is private: your codebase, your ticket history, your contracts. None of that was in the pretraining data and none of it ever will be.

Freshness. A document you edited this morning is retrievable this afternoon. Nothing about the model has to change for that to be true — which is exactly why RAG became the default answer rather than retraining.

Citability. This one is underrated and, for a lot of real deployments, is the actual product. Because you chose the chunks, you know which documents produced the answer, and you can show them to the reader. That doesn't make the model truthful — more on that below — but it converts "trust me" into "here's the paragraph," which is the difference between a demo and something a compliance team will sign. It's also the most practical mitigation available for the failure mode we called hallucination: you can't stop a model from producing fluent wrong text, but you can put the source next to it and make the wrongness checkable.

Economics. Re-embedding a changed document costs a fraction of a cent. Retraining a model to absorb it does not.

Everything hard here is a retrieval problem

Now count the steps in that pipeline again and ask which of them is new.

Collect documents. Split them. Index them. Match a query against the index. Rank the matches. Decide how many to keep. Assemble them into a response. Every one of those is a problem search engineers have been arguing about since the 1970s. Exactly one thing is new — the last step reads the results and writes prose instead of showing you ten blue links.

This is not a cute framing. It predicts where your time will go.

Chunking has no principled answer. You are cutting continuous documents into pieces sized for an embedding model, and every cut is lossy. A chunk that reads "it does not support this configuration" is useless when the antecedent of it was three paragraphs up and got severed. Anthropic's contextual retrieval writeup is the clearest published measurement of this: prepending 50–100 tokens of document context to each chunk before embedding cut the top-20 retrieval failure rate from 5.7% to 3.7% — a 35% reduction, purely from fixing what chunking destroyed.

Read the rest of that result, though, because it's the thesis in miniature. Adding BM25 — a lexical ranking function whose core formulation dates to 1994 — on top of contextual embeddings took failure from 5.7% down to 2.9%, a 49% reduction. Adding a reranking pass took it to 1.9%, a 67% reduction. The two biggest wins in a modern retrieval stack came from a thirty-year-old keyword algorithm and a classical ranking step.

Recall is a hard ceiling. If the answer isn't in the chunks you retrieved, no prompt engineering rescues it. This is the single most important operational fact about RAG and it gets lost constantly, because the failure doesn't look like a search failure — it looks like the model being stupid.

Vectors are bad at some of the things you'll search for. The embeddings post made this point in the abstract; here's the concrete version. Dense retrieval genuinely beats keyword search on natural-language questions — Karpukhin et al. measured 9–19 points of top-20 accuracy over BM25 — but it is measurably worse at exact tokens: error codes, SKUs, function names, version numbers, surnames. Semantic similarity is the wrong tool for finding the one document containing ERR_TLS_CERT_ALTNAME_INVALID. Hybrid search is the default in serious systems for exactly this reason, and "hybrid" here means "we kept the 1994 algorithm."

Freshness and permissions are infrastructure, not intelligence. Who is allowed to see this chunk? When was it reindexed? What happens to the index when a document is deleted? These questions have no AI content whatsoever, and they will eat more of your quarter than prompt design will.

The uncomfortable summary: if you're hiring for this, the job posting probably asks for LLM experience. It should ask for search experience.

How it fails

Because there are two subsystems, there are two failure modes, and the first diagnostic move is always to determine which one you have.

Retrieval failure: the right chunk was never in the set. Generation failure: it was there and the model ignored it, contradicted it, or padded around it. These have completely different fixes, and conflating them is how teams end up rewriting prompts for three weeks against a chunking bug. Check the retrieved set before you touch the prompt. Most of the time "the model hallucinated" is a recall bug wearing a costume.

Beyond that split, four things bite reliably:

More context is not better context. Nelson Liu et al.'s Lost in the Middle (TACL) found a U-shaped accuracy curve: models use information best at the beginning and end of the input and measurably worse in the middle. Stuffing twenty chunks in because you have the window for it can bury the good one in the sag.

The relationship between retrieval quality and answer quality is not monotone. This is the genuinely strange result, from Cuconasu et al.'s The Power of Noise at SIGIR 2024: highly-ranked documents that are related-but-not-relevant — near misses, the kind a good retriever produces — hurt accuracy, while injecting random documents improved it by up to 35%. You do not need a theory of why to take the lesson, which is that you cannot reason your way to the right retrieval configuration analytically. The previous post's argument about prompting applies here verbatim, and for the same reason.

Sources disagree, and the model picks one. Two versions of a policy document, one superseded. Both retrieved. The answer is confident and cites the wrong one. Corpus hygiene is part of the system, not a prerequisite you get to assume.

Grounding is not truth. RAG guarantees the answer was conditioned on the retrieved text. It guarantees nothing about whether that text is correct. Retrieve garbage and you get cited garbage, which is worse than uncited garbage because it looks like diligence.

How you'd actually measure it

Two failure modes means two eval sets. Running one is the most common mistake after not running any.

Retrieval evaluation needs no model at all. Build a set of questions labeled with which documents should answer them — fifty is enough to catch a bad chunking change — and measure recall@k (was the right chunk in the top k?) and nDCG (was it near the top?). This is cheap, deterministic, and fast enough to run on every commit. It is also the only measurement that tells you where your ceiling is.

Answer evaluation is the fuzzy half, and it's what frameworks like RAGAS exist to systematize. The four metrics worth internalizing split cleanly along the same seam: context precision and context recall grade the retrieved set, while faithfulness (are the answer's claims actually supported by the retrieved text?) and answer relevance (does it address the question?) grade the generation. Faithfulness is the one to watch — an unfaithful answer over a perfect retrieval is a prompt problem, and a faithful answer over an empty retrieval is a confident "I don't know," which is a success.

That's the same discipline the last post argued for, arriving from a different direction. You cannot predict the effect of a chunk-size change, a k change, or an embedding-model swap. You can only measure it.

"Is RAG dead?"

You will see this claim a lot, and it deserves a serious answer rather than a dismissal, because the best evidence for it is very good.

In 2025, Anthropic removed vector search from Claude Code. Boris Cherny, who built it, explained why:

Early versions of Claude Code used RAG + a local vector db, but we found pretty quickly that agentic search generally works better. It is also simpler and doesn't have the same issues around security, privacy, staleness, and reliability.

Take that seriously. For a codebase, grep is a better retriever than an embedding index: exact symbol matching, never stale, no chunking heuristic to tune, no index to rebuild, no copy of your proprietary source sitting in a vector database. The tool became more capable and an entire subsystem stopped earning its keep.

Now notice what did not happen. They did not stop retrieving. They replaced an approximate-nearest-neighbour index with a lexical search tool and handed query formulation to the model instead of computing it from a single embedding. Agentic search is retrieval with the model acting as its own query planner — iterating, refining, and reading, which is what a developer does and what a one-shot top-k lookup never could. The retrieval method got replaced. The retrieval step is load-bearing as ever. That's the thesis, not a counterexample to it.

The long-context argument has a similar shape and a similar answer. Million-token windows are real; so is prompt caching. But three things don't go away:

Corpora are bigger than windows, by orders of magnitude. Your company wiki is not a million tokens. It's four hundred million.

Tokens cost money and latency, both of which scale with how much you stuff in. Retrieval is also a cost-control mechanism.

Long windows degrade before they fill. Chroma's Context Rot report (Hong, Troynikov, and Huber, July 2025) evaluated 18 frontier models and found that performance drops as input length grows "often in surprising and non-uniform ways" — well before the documented limit. A 1M-token window is not a promise of reasoning across 1M tokens.

The academic literature is genuinely split, which is itself informative. Xinze Li et al.'s Long Context vs. RAG found long context generally ahead on Wikipedia-style QA while RAG held an edge on dialogue-based and general queries — and, in the detail that matters most here, that summarization-based retrieval performed comparably to long context while chunk-based retrieval lagged behind. When the retrieval strategy changes the verdict, the retrieval strategy was the variable all along.

So: naive fixed-size chunking with top-k vector search is being absorbed, deservedly, the way the 2023 prompt tricks were absorbed. The decision about what goes in the window is permanent. It's the only decision there has ever been.

Honest limits

Four things RAG does not do, stated plainly so it doesn't become the answer to every question.

RAG does not raise the capability ceiling. It changes what's in the window, not what the model can do with what's in the window. If the task requires reasoning the model can't perform, perfect retrieval produces a well-sourced wrong answer.

RAG does not teach behaviour. Format, tone, house style, the tacit judgment of a domain — retrieval can show examples of these, but it doesn't change the model's disposition toward them. That boundary is exactly what the next post is about.

Retrieved documents are an attack surface. Everything retrieved lands in the same window as your instructions, with no privilege separation between them — because, as established, there isn't any. Greshake et al. named this indirect prompt injection: an attacker who can write into your corpus can write into your prompt. If your corpus includes support tickets, public wiki pages, or fetched web content, that attacker is anyone. A prompt is not a security boundary, and a retrieved prompt is a prompt someone else wrote.

Retrieval inherits your permission model, or invents one badly. The moment two users are allowed to see different documents, chunk-level access control becomes your problem, and the vector index is a spectacularly convenient place to leak from.

Where this leaves us

The frozen function still hasn't changed. Prompting was writing to the window by hand; retrieval is writing to it programmatically, with a search engine deciding what goes in. Both are answers to the same question this whole section is built around — what's in the window, and how did it get there?

Which sets up the obvious next question. Retrieval puts knowledge in the window at query time. Fine-tuning puts behaviour in the weights ahead of time. Prompting arranges what you already have. These get pitched as competing options and they mostly aren't — they fix different things. The next post makes that comparison honestly, including the cases where the answer really is "fine-tune," which are rarer and more specific than the marketing suggests.

Then tools, MCP, and multimodality — three more ways of getting something into that window that you didn't type.

Further reading

The original idea

• Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020) — the paper that named it.
• Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering (EMNLP 2020) — dense retrieval beating BM25 by 9–19 points of top-20 accuracy.
• Robertson & Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond — the algorithm still doing half the work.

Why retrieval is the hard part

• Anthropic (2024). Introducing Contextual Retrieval — 35% / 49% / 67% failure-rate reductions, stacked.
• Cuconasu et al. (2024). The Power of Noise: Redefining Retrieval for RAG Systems (SIGIR 2024) — near-miss documents hurt; random ones helped by up to 35%.
• Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts (TACL) — the U-shaped curve.

Measuring it

• Es et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation — reference-free metrics split across retrieval and generation.

The long-context argument

• Hong, Troynikov & Huber (2025). Context Rot: How Increasing Input Tokens Impacts LLM Performance — 18 models, non-uniform degradation well before the limit.
• Li et al. (2024). Long Context vs. RAG for LLMs: An Evaluation and Revisits — where each wins, and why chunk-based retrieval lags summarization-based.
• Cherny (2025). On why Claude Code dropped RAG for agentic search.

Security

• Greshake et al. (2023). Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.