Retrieval-Augmented Generation

A unit on what happens after retrieval when the reader is a language model. Open-domain question answering and its measures, the RAG models and REALM with the retrieved passage as a hidden variable, Fusion-in-Decoder and the cost of reading many passages, nearest neighbor language models, passages and chunking, assembling the context, retrieval driven by the model, attribution and citation, and the evaluation of retrieval-augmented systems. Each article says what it proves, what it cites and what its experiment measured.

The first three units of this collection built retrievers: dense ones that compare meanings, indexes that search vectors quickly, and re-rankers and fusion that order a short list well. Each of them ended with a ranked list of passages, judged by measures that assume a person reads the list from the top. This unit is about the case in which the reader is a language model. The model reads the passages it is given and writes an answer, so what matters is no longer only whether good passages are near the top. It is also whether the model can use them, whether its answer says what the passages say, and how much it costs to read them. The questions are old ones of information retrieval, about what counts as relevant and how a system's output is judged, asked of a new reader.

The unit has four parts. The first is the design itself: open-domain question answering, the task on which retrieval and reading were joined and measured at TREC-8; the RAG models and REALM, which train a retriever from answers alone by treating the passage as a hidden variable; Fusion-in-Decoder, which lets one model read a hundred passages at once; and nearest neighbor language models, which retrieve single next tokens instead of passages. The second is the preparation of what is read: how documents are cut into passages, and how the retrieved passages are put into a model's context, in what number and order. The third is retrieval that the model drives, by writing its own queries or calling a search tool. The fourth is checking the output: whether each statement is supported by a cited source, and how a retrieval-augmented system as a whole is evaluated.

What the unit assumes

The earlier units of this collection: Text Embeddings and Dense Retrieval, in particular Dense Passage Retrieval, whose retriever most of the systems here start from; Vector Search for the index; and Neural Ranking and Hybrid Retrieval, whose Result Diversification ends with the question this unit begins with. From the course: the measures of ranked retrieval evaluation, test collections and relevance, and document delineation for what counts as a unit of retrieval. The transformer is assumed as the Language Models from Scratch collection builds it, and the site's own search, with its JSON API for assistants, is described in How This Search Works.

What is proved, cited and measured

This unit has less mathematics than the others and more defined measures; the reference style is kept by exact definitions, proofs of what can be proved, and examples computed in full. Derived or proved on the pages, among other results: that exact match never exceeds token F1 and that F1 is a Dice coefficient of token bags; that no extractive reader's exact match can exceed the rate at which its passages contain the answer; that the RAG-Sequence and RAG-Token likelihoods are probability distributions, how each looks when written one token at a time, and that for two tokens their difference is minus a covariance, and RAG-Sequence gives the higher probability whenever the passages agree in their support for every token; that the gradient of either with respect to the retrieval scores is posterior minus prior (summed over tokens for RAG-Token), which explains why retrieval, once a generator ignores its passages, receives no signal to recover; the cost of reading $k$ passages separately or concatenated, and that Fusion-in-Decoder ignores the order of its passages but counts a duplicate twice. The later articles add the interpolation of a nearest neighbor language model, citation precision and recall on a worked case, and agreement statistics for model judges.

Cited and not proved: every published measurement of effectiveness, each with the table or figure it comes from. Several subjects of the unit were changing quickly as of November 2026, in particular retrieval driven by the model, the placement of passages in long contexts and the evaluation of generated answers; the articles on them are confined to named, published methods and say when they were written.

The test beds are of three kinds. Public question-answering data: the TREC-8 question answering track's questions, answer patterns and judgments, published by NIST, and the open version of Natural Questions. This site's own pages and its judged queries, as earlier units used them. And, where a result needs a large generator, small runs of a hosted language model recorded with their prompts, model identifier and outputs; such a run is an observation of one model at one date, not an experiment repeated over seeds, and the articles that report one say so.

Notation

The unit keeps the collection's notation. A query or other input is $q$ and a passage $p$; $d$ is used for whole documents. The score of a passage is $s(q, p)$, and the retrieved list is $p_1, \dots, p_k$, the $k$ highest-scoring passages of a collection of $N$. Because lowercase $p$ is a passage, probabilities are written with a capital $P$: the retriever's distribution over the retrieved passages is $P_\eta(p \mid q)$, a softmax of the scores over the top $k$, and the generator's is $P_\theta$. An output is a token sequence $y = (y_1, \dots, y_T)$ with prefix $y_{<t}$. A question's gold answers are a set $A$ and a predicted answer is $\hat a$; exact match and F1 are the measures defined in Open-Domain Question Answering. "Top-$k$ accuracy" is the share of questions with a gold answer in the top $k$ passages. A transformer has $\ell$ blocks of width $m$, and a text has $L$ tokens. Logarithms are natural, $\ln$.

Reading retrieved text

  • Open-Domain Question Answering: the retriever and reader design from the TREC-8 question answering track (Voorhees, 1999) to DrQA (Chen et al., 2017) and the dense retrievers; exact match and F1 defined as SQuAD's script computes them, with their properties proved; all 37,927 judged answer strings of TREC-8 scored by assessors, answer patterns, exact match and F1; and the bound, proved, that an extractive reader cannot exceed its passages' answer containment, read against the published tables.
  • The RAG Models: RAG-Sequence and RAG-Token (Lewis et al., 2020) with the retrieved passage as a hidden variable, computed in full on a two-passage, three-token example; what can be proved about the two likelihoods; the retriever's gradient as posterior minus prior and why the retriever gets no signal once the generator ignores its passages; REALM; parametric and non-parametric memory, and the published results.
  • Fusion-in-Decoder and Reading Many Passages: encoding passages separately and fusing them in the decoder; the cost against concatenation, derived and evaluated for T5's sizes; why decoding is slow; order invariance and duplicates, proved; scaling with the number of passages as reported; the reader's attention as a teacher of the retriever, and Atlas in outline.
  • Nearest Neighbor Language Models: retrieval at the level of single tokens, the other end of the design space; kNN-LM's datastore of contexts and next tokens and its interpolation formula, worked on a small GPT trained here; RETRO in outline.

Preparing what is read

  • Passages and Chunking: what the unit of retrieval should be; passage retrieval before language models, from fixed windows to discourse units and TextTiling; chunk size and overlap as measured parameters on this site's pages; adding document context to a passage before indexing.
  • Assembling the Context: how many passages, in what order and in what budget; the position effect of Liu et al.; redundancy; the in-context design, retrieved text placed before a frozen model, as opposed to a jointly trained one.

Retrieval driven by the model

  • Retrieval in a Loop: queries written by the model, from hypothetical documents (HyDE) to retrieval interleaved with reasoning (IRCoT), retrieval the model decides to do (Self-RAG) and search as a tool call, with this site's search API as the worked case.

Checking the output

  • Attribution and Citation: what it means for a generated statement to be supported by a source; citation precision and recall as defined by Gao et al., computed on a small judged example; the limits of automatic checks.
  • Evaluating Retrieval-Augmented Systems: separating retrieval error from generation error; provenance-aware measures; judging relevance and answers with language models and how far that can be trusted, with this site's own model-judged evaluation as the worked case.

The next unit, Efficient Query Processing, returns to the engine underneath all of this: answering a query without scoring every posting.

References

  • Chen, D., Fisch, A., Weston, J., and Bordes, A. (2017). Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 1870–1879. https://doi.org/10.18653/v1/P17-1171. arXiv:1704.00051. record · 3 pages
  • Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33, 9459–9474. arXiv:2005.11401. record · 6 pages
  • Voorhees, E. M. (1999). The TREC-8 question answering track report. In Proceedings of the Eighth Text REtrieval Conference (TREC-8), NIST Special Publication 500-246, 77–82. record · 5 pages