Lewis et al. (2020) gave their model the single word "Hemingway" and asked it to write a quiz clue to which that word is the answer. It retrieved five Wikipedia passages, and wrote a clue that names two of his novels, "The Sun Also Rises" and "A Farewell to Arms". One passage mentioned the first novel and another the second. The paper's Figure 2 shows which passage the model leaned on as it wrote each word: the second passage while it began the first title, the first passage while it began the other. Nobody had told the model which passage held which title; it had been trained on pairs of answers and clues only. How can a generator learn to use retrieved passages, and a retriever learn which passages to retrieve, when no training example says which passage is the right one?
Retrieval-augmented generation, in the sense of Lewis et al. (2020), is a model that writes its output conditioned on passages retrieved for the input, and that is trained end to end by treating the retrieved passage as an unobserved, or latent, variable. Its knowledge is held in two places: in the weights of the generator, which the paper calls the parametric memory, and in an index of text, the non-parametric memory. REALM (Guu et al., 2020) used the same latent-variable formulation a few months earlier to pre-train a retriever. Later systems also place retrieved text in the prompt of a language model that is not trained with the retriever at all, which Ram et al. (2023) call In-Context RALM: "leaving the LM architecture unchanged and prepending grounding documents to the input, without any further training of the LM". This article is about the original models, in which the passage is a variable to be summed over; Assembling the Context treats the in-context design.
Here is the route. First we define the two memories and the setting. Then we write down the model: a retriever that puts a probability on each passage, and two ways of combining it with a generator, RAG-Sequence and RAG-Token, computed in full on a small example. Next we prove what can be proved about the two likelihoods: that each is a probability distribution, how each looks when written one token at a time, and when one is larger than the other. Then comes training, where the gradient with respect to the retriever turns out to be posterior minus prior, which is also REALM's update and the reason retrieval can collapse. Finally we quote what the papers report. The article assumes Open-Domain Question Answering for the task and its measures, Dense Passage Retrieval for the retriever, and the softmax and transformer of the Language Models from Scratch collection. The numbers are computed by tools/experiments/rag/rag-sequence-and-rag-token.py, in exact rational arithmetic.
Two memories
A language model fine-tuned to answer questions can answer some of them with no retrieval at all, from what its weights absorbed in pre-training. Lewis et al. (2020) call such a model's knowledge "parametric" and such question answering "closed-book". REALM's introduction states the cost of keeping knowledge there: it is stored "implicitly in the parameters of the underlying neural network", which makes it "difficult to determine what knowledge is stored in the network and where", and "storage space is limited by the size of the network" (Guu et al., 2020, §1). Lewis et al. (2020, §1) add that such models "cannot easily expand or revise their memory" and "may produce 'hallucinations'".
A retrieval-augmented model keeps a second memory outside the weights: a collection of passages, indexed for search. In RAG it is all of English Wikipedia, cut into disjoint 100-word passages (about 21 million), each encoded by DPR's passage encoder into a vector and stored in a FAISS index (§3). "We refer to the document index as the non-parametric memory", and to the generator's parameters "as the parametric memory" (§2.2, §2.3). The weights are fixed in number once the model is built; the index has no trained parameters and grows with the collection.
The second memory can be read and replaced. Lewis et al. (2020, §4.5) built a second index from the December 2016 Wikipedia dump of DrQA and asked their model "Who is {position}?" for 82 world leaders who had changed between 2016 and 2018. With the 2016 index it answered 70% of the 2016 leaders correctly, and with the 2018 index 68% of the 2018 leaders; with the indexes swapped, 12% and 4%. The weights were the same in all four cases.
The model
A retriever that puts a probability on passages
RAG's retriever is DPR's bi-encoder: a query encoder $E_Q$ and a passage encoder $E_P$, each a BERT-base network read at its [CLS] vector, and the score of passage $p$ for input $q$ is the dot product $s(q, p) = E_Q(q) \cdot E_P(p)$ (Lewis et al., 2020, §2.2). Since the input need not be a question, the paper writes it $x$; we keep the collection's $q$. The retriever turns scores into a probability distribution with a softmax,
$$P_\eta(p \mid q) \propto e^{s(q, p)},$$
where $\eta$ stands for the retriever's parameters. Summing $e^{s(q,p)}$ over 21 million passages for every input is out of the question, and the approximation below is reasonable, as Guu et al. (2020, §3.3) put it for the same model, "if most documents have near zero probability". The model keeps only the $k$ passages with the highest scores, $p_1, \dots, p_k$, found by maximum inner product search, and normalizes over them:
$$P_\eta(p_i \mid q) = \frac{e^{s(q, p_i)}}{\sum_{j=1}^{k} e^{s(q, p_j)}}, \qquad i = 1, \dots, k.$$
This is the "top-K approximation" of the paper (§2.1). We call $P_\eta(p_i \mid q)$ the prior over passages: the retriever's belief before any output is seen. We write $\pi_i = P_\eta(p_i \mid q)$ when the input is fixed.
A generator that reads one passage
The generator is BART-large, a pre-trained sequence-to-sequence transformer with about 400 million parameters; the input and the passage are simply concatenated to form its input (§2.3). For an output of $T$ tokens $y = (y_1, \dots, y_T)$ it gives, for each passage $p$, the probability of each token given the earlier ones:
$$P_\theta(y_t \mid q, p, y_{<t}), \qquad y_{<t} = (y_1, \dots, y_{t-1}),$$
where $\theta$ stands for its parameters. Write $a_t(i) = P_\theta(y_t \mid q, p_i, y_{<t})$ for the probability of the $t$-th target token when the generator reads passage $p_i$, and $G_i = \prod_{t=1}^{T} a_t(i)$ for the probability of the whole output given that passage.
Nothing in the training data says which passage the output should be read from. That is what makes the passage latent: it is summed over, weighted by the prior.
Two ways to sum over passages
The two RAG models differ in when the sum is taken (Lewis et al., 2020, §2.1).
RAG-Sequence "uses the same retrieved document to generate the complete sequence". One passage is drawn for the whole output:
$$P_{\text{Seq}}(y \mid q) = \sum_{i=1}^{k} P_\eta(p_i \mid q) \prod_{t=1}^{T} P_\theta(y_t \mid q, p_i, y_{<t}) = \sum_{i=1}^{k} \pi_i\, G_i .$$
RAG-Token "can draw a different latent document for each target token". The sum is inside the product:
$$P_{\text{Tok}}(y \mid q) = \prod_{t=1}^{T} \sum_{i=1}^{k} P_\eta(p_i \mid q)\, P_\theta(y_t \mid q, p_i, y_{<t}) = \prod_{t=1}^{T} \sum_{i=1}^{k} \pi_i\, a_t(i).$$
In words: RAG-Sequence is a mixture of $k$ complete sequence models, one per passage. RAG-Token is a single sequence model whose every next-token distribution is a mixture of $k$ next-token distributions. Both use the same retriever and the same generator; only the place of the sum differs. The paper notes that for a target of one token, as in classification, the two are equal (§2.1), which the formulas show with $T = 1$.
The running example
Here is the Hemingway case reduced to numbers small enough to follow. The numbers are invented to show the mechanics; they are not taken from the paper. The input $q$ asks for the novel, the target is three tokens, "Sun Also Rises", and two passages have been retrieved:
- $p_1$, a biography that names other works but not this one, with the higher retrieval score;
- $p_2$, a passage that names the novel.
The retriever's scores differ by $\ln 3$, so the prior is $\pi = (3/4, 1/4)$, since $e^{\ln 3}/(e^{\ln 3} + 1) = 3/4$. The generator's token probabilities are:
| Passage | $a_1$ ("Sun") | $a_2$ ("Also", after "Sun") | $a_3$ ("Rises", after "Sun Also") | $G_i$ |
|---|---|---|---|---|
| $p_1$ | 1/10 | 4/5 | 9/10 | 0.072 |
| $p_2$ | 3/5 | 9/10 | 9/10 | 0.486 |
Reading $p_1$, the generator is unlikely to start with "Sun", but once it has, it completes the title from its own weights, which is the behavior the paper describes in its Figure 2: after the first token of each title, the posterior over documents "flattens", suggesting that "the model's parametric knowledge is sufficient to complete the titles" (§4.3).
RAG-Sequence: $P_{\text{Seq}} = \tfrac34 \cdot 0.072 + \tfrac14 \cdot 0.486 = 0.054 + 0.1215 = 0.1755$.
RAG-Token: the three mixtures are $\tfrac34 \cdot \tfrac1{10} + \tfrac14 \cdot \tfrac35 = 0.225$, then $\tfrac34 \cdot \tfrac45 + \tfrac14 \cdot \tfrac9{10} = 0.825$, then $0.9$, and $P_{\text{Tok}} = 0.225 \times 0.825 \times 0.9 = 0.16706$ (exactly $0.1670625$).
The two models assign different probabilities to the same output from the same parts. Here RAG-Sequence's is larger. Why, and whether it always is, is the subject of the next section.
What can be proved about the two likelihoods
Both are probability distributions
Proposition 1. Fix the input $q$, the $k$ retrieved passages and a length $T$. If for each passage and each prefix the generator's next-token probabilities sum to 1 over the vocabulary, then $\sum_y P_{\text{Seq}}(y \mid q) = 1$ and $\sum_y P_{\text{Tok}}(y \mid q) = 1$, the sums running over all sequences of $T$ tokens.
Proof. For each passage $p_i$, the chain rule makes $G_i(y) = \prod_t a_t(i)$ a probability distribution over sequences: summing over $y_T$ first gives 1 for every prefix, then summing over $y_{T-1}$, and so on down to $y_1$. Hence $$\sum_y P_{\text{Seq}}(y \mid q) = \sum_y \sum_i \pi_i G_i(y) = \sum_i \pi_i \sum_y G_i(y) = \sum_i \pi_i = 1.$$ For RAG-Token, the factor $\sum_i \pi_i\, P_\theta(y_t \mid q, p_i, y_{<t})$ is, for each prefix, a convex combination of distributions over $y_t$, hence a distribution over $y_t$. A product of conditional distributions over successive tokens sums to 1 over all sequences by the same backward summation. $\blacksquare$
In practice outputs end with an end-of-sequence token rather than at a fixed length. Call such an output finished; no finished output is a prefix of another. Then the probabilities of all finished outputs sum to at most 1, with equality when the model stops with probability 1. For one passage: the network gives a next-token distribution after every prefix, the end token included, so the probability of a finished output $y$ of length $m \le n$ equals the total probability of the length-$n$ sequences that begin with $y$. These sets of sequences are disjoint for different finished outputs, so the finished outputs of length at most $n$ have total probability at most that of all length-$n$ sequences, which is 1; let $n$ grow. RAG-Sequence, a mixture with weights $\pi_i$, inherits the bound, and RAG-Token is an ordinary autoregressive model whose next-token distribution is $\sum_i \pi_i P_\theta(\cdot \mid q, p_i, y_{<t})$, to which the same argument applies.
Each written one token at a time
RAG-Token is already in the form of an ordinary autoregressive model: its probability of the next token is $\sum_i \pi_i\, a_t(i)$, which is why it can be decoded with a standard beam search (§2.5). RAG-Sequence can be put in the same form, and the comparison explains the difference between the models.
From here on, every output considered has positive probability under the model, so that the divisions below are defined.
Proposition 2 (RAG-Sequence token by token). Let $P_{\text{Seq}}(y_{\le t} \mid q) = \sum_i \pi_i \prod_{s \le t} a_s(i)$ be the probability of the first $t$ tokens. Then $$\frac{P_{\text{Seq}}(y_{\le t} \mid q)}{P_{\text{Seq}}(y_{<t} \mid q)} = \sum_{i=1}^{k} P(p_i \mid q, y_{<t})\; a_t(i), \qquad P(p_i \mid q, y_{<t}) = \frac{\pi_i \prod_{s<t} a_s(i)}{\sum_j \pi_j \prod_{s<t} a_s(j)} .$$
Proof. Divide $\sum_i \pi_i \prod_{s \le t} a_s(i) = \sum_i \big(\pi_i \prod_{s<t} a_s(i)\big)\, a_t(i)$ by $\sum_j \pi_j \prod_{s<t} a_s(j)$. $\blacksquare$
The weight $P(p_i \mid q, y_{<t})$ is the posterior probability of passage $p_i$ after the first $t - 1$ tokens, by Bayes' rule with prior $\pi$ and likelihood $\prod_{s<t} a_s(i)$. So the two models generate the next token from a mixture over passages with different weights:
- RAG-Sequence weights the passages by their posterior, which is updated by every token generated so far;
- RAG-Token weights them by the prior at every step, which never moves.
A passage that explained the earlier tokens well gains weight in RAG-Sequence for the rest of the output. In RAG-Token each token starts afresh from the retriever's opinion.
In the running example the posterior on $p_2$ is $1/4$ before any token, $2/3$ after "Sun" and $9/13 = 0.692$ after "Sun Also"; it stays at 0.692 after "Rises", since both passages give that token 9/10. RAG-Sequence's conditional probability of "Also" is therefore $\tfrac13 \cdot \tfrac45 + \tfrac23 \cdot \tfrac9{10} = 0.867$, against RAG-Token's 0.825, which uses the prior weights. The product of RAG-Sequence's three conditionals, $0.225 \times 0.867 \times 0.9$, is again 0.1755, as Proposition 2 requires.
RAG-Token has a posterior too, but a per-token one. Given the token actually generated, the weight passage $p_i$ had in producing it is $$\rho_t(i) = \frac{\pi_i\, a_t(i)}{\sum_j \pi_j\, a_t(j)},$$ which is what the paper's Figure 2 plots for each generated token. In the example $\rho_t(2)$ is $2/3$ for "Sun", $3/11 = 0.273$ for "Also" and $1/4$ for "Rises": high where the passage made the difference, and back near the prior where it did not.
When one model gives the higher probability
Proposition 3 (sufficient conditions for equality). $P_{\text{Seq}}(y \mid q) = P_{\text{Tok}}(y \mid q)$ if any of the following holds: $T = 1$; $k = 1$; the prior puts all its mass on one passage; or, for $y$, the generator's token probabilities do not depend on the passage, $a_t(i) = a_t$ for all $i$ and $t$.
Proof. With $T = 1$ both are $\sum_i \pi_i a_1(i)$. With $k = 1$, or with $\pi_i = 1$ for one $i$, both are $\prod_t a_t(i)$ for that $i$. If $a_t(i) = a_t$ for all $i$, then $P_{\text{Seq}} = \sum_i \pi_i \prod_t a_t = \prod_t a_t$ and $P_{\text{Tok}} = \prod_t \sum_i \pi_i a_t = \prod_t a_t$. $\blacksquare$
Outside these cases the two differ, and for two tokens the difference has an exact form. Treat the passage index $I$ as a random variable with distribution $\pi$, and the token probabilities $a_1(I)$, $a_2(I)$ as random variables through it.
Proposition 4 (two tokens). For $T = 2$, $$P_{\text{Tok}}(y \mid q) - P_{\text{Seq}}(y \mid q) = -\operatorname{Cov}_\pi\!\big(a_1(I), a_2(I)\big).$$
Proof. $P_{\text{Tok}} = \big(\sum_i \pi_i a_1(i)\big)\big(\sum_i \pi_i a_2(i)\big) = \mathbb{E}_\pi[a_1(I)]\, \mathbb{E}_\pi[a_2(I)]$ and $P_{\text{Seq}} = \sum_i \pi_i a_1(i) a_2(i) = \mathbb{E}_\pi[a_1(I)\, a_2(I)]$. Their difference is $\mathbb{E}[a_1]\mathbb{E}[a_2] - \mathbb{E}[a_1 a_2] = -\operatorname{Cov}(a_1, a_2)$, by the definition of covariance in Expectation, Variance and Covariance. $\blacksquare$
The sign of the covariance says whether the passages that support one token also support the other. With two passages it is exact: $\operatorname{Cov}_\pi(a_1, a_2) = \pi_1 \pi_2 \big(a_1(1) - a_1(2)\big)\big(a_2(1) - a_2(2)\big)$, positive when the same passage gives both tokens the higher probability, so that RAG-Sequence, which commits to one passage, wins, and negative when each passage favors a different token, so that RAG-Token, which can switch, wins. With more passages the covariance mixes many such comparisons and the description is only a tendency; Proposition 4b below gives a condition under which it is exact. The Hemingway clue, one title from each of two passages, is the second kind; the paper reports that RAG-Token did better on that task (Table 2) and gives this reason: "it can generate responses that combine content from several documents" (§4.3).
For three tokens the same computation gives an exact identity. Write $\mu_t = \mathbb{E}_\pi[a_t(I)]$ and $\kappa = \mathbb{E}_\pi\big[(a_1 - \mu_1)(a_2 - \mu_2)(a_3 - \mu_3)\big]$, the joint third central moment. Expanding the product inside $\kappa$ and taking expectations term by term gives $\kappa = \mathbb{E}[a_1 a_2 a_3] - \mu_1 \operatorname{Cov}(a_2, a_3) - \mu_2 \operatorname{Cov}(a_1, a_3) - \mu_3 \operatorname{Cov}(a_1, a_2) - \mu_1 \mu_2 \mu_3$, since $\mathbb{E}[a_s a_t] = \operatorname{Cov}(a_s, a_t) + \mu_s \mu_t$. With $P_{\text{Tok}} = \mu_1\mu_2\mu_3$ and $P_{\text{Seq}} = \mathbb{E}[a_1 a_2 a_3]$, this is $$P_{\text{Tok}} - P_{\text{Seq}} = -\big[\kappa + \mu_1 \operatorname{Cov}(a_2, a_3) + \mu_2 \operatorname{Cov}(a_1, a_3) + \mu_3 \operatorname{Cov}(a_1, a_2)\big].$$ The pairwise covariances and the third moment can have either sign, and the random cases below show that their sum, and so the difference, takes either sign. The script checks the identity on 2,000 random cases in exact arithmetic, and draws 10,000 random cases with two passages, three tokens and probabilities in tenths (seed 20261118): RAG-Token is larger in 5,423, RAG-Sequence in 4,235, and they are equal in 342.
One condition does fix the sign for any number of tokens and passages.
Proposition 4b (passages that agree). Suppose the passages can be ordered so that every token's probability $a_t(i)$ is nondecreasing along the order, for every $t$: the passages that support one token more support every token more. Then $P_{\text{Seq}}(y \mid q) \ge P_{\text{Tok}}(y \mid q)$.
Proof. First, Chebyshev's sum inequality: if $f$ and $g$ are nondecreasing along the order, then $\mathbb{E}_\pi[fg] \ge \mathbb{E}_\pi[f]\,\mathbb{E}_\pi[g]$. Indeed $$\sum_i \sum_j \pi_i \pi_j \big(f(i) - f(j)\big)\big(g(i) - g(j)\big) = 2\,\mathbb{E}[fg] - 2\,\mathbb{E}[f]\,\mathbb{E}[g],$$ as multiplying out and using $\sum_i \pi_i = 1$ shows, and every term on the left is nonnegative, since $f(i) - f(j)$ and $g(i) - g(j)$ have the same sign or one is zero. Second, a product of nonnegative nondecreasing functions is nondecreasing: for $i$ before $j$, $f(i) g(i) \le f(j) g(i) \le f(j) g(j)$. Now let $F_t = \prod_{s \le t} a_s$, which is nonnegative and nondecreasing. By the inequality, $\mathbb{E}[F_t] = \mathbb{E}[F_{t-1}\, a_t] \ge \mathbb{E}[F_{t-1}]\, \mathbb{E}[a_t]$, and by induction $\mathbb{E}[F_T] \ge \prod_t \mathbb{E}[a_t]$, which is $P_{\text{Seq}} \ge P_{\text{Tok}}$. $\blacksquare$
The running example satisfies the condition with the order $p_1, p_2$: $1/10 < 3/5$, $4/5 < 9/10$, $9/10 = 9/10$. That is why RAG-Sequence wins there. The script checks Proposition 4b on 20,000 random cases with up to five passages and five tokens, in exact arithmetic, and finds no violation. Without the condition either model can give the higher probability, as the random count above shows.
Example (the more flexible model need not fit better). "RAG-Token is the more flexible model, so it should always fit better." The running example is a counterexample: with one passage supporting the whole title, RAG-Sequence gives the target 0.1755 and RAG-Token 0.16706. RAG-Token's freedom to switch passages is also a freedom to switch to the wrong one at every token, and it pays for that freedom when one passage explains the whole output, as here.
Training: learning to retrieve from answers
RAG is trained on pairs of input and output, $(q_j, y_j)$, by minimizing the negative log-likelihood of each output, $-\ln P(y_j \mid q_j)$, with either model's $P$ (§2.4). The passage encoder $E_P$ is kept fixed, because updating it would require re-encoding and re-indexing all 21 million passages; the query encoder $E_Q$ and the generator are trained. The question this section answers is what that objective does to the retriever.
Here is the idea in one sentence: the gradient raises the score of each retrieved passage in proportion to how much more probable it makes the output than the retriever expected. The derivation makes "how much more probable" precise.
The softmax derivative. For $\pi_i = e^{s_i}/\sum_j e^{s_j}$ with $s_i = s(q, p_i)$, $$\frac{\partial \pi_i}{\partial s_j} = \pi_i\, (\delta_{ij} - \pi_j),$$ where $\delta_{ij}$ is 1 if $i = j$ and 0 otherwise. (Differentiate the quotient: the numerator contributes $\delta_{ij} e^{s_i}$ and the denominator $-e^{s_i} e^{s_j}/(\sum e)^2$.)
Proposition 5 (the retriever's gradient). For RAG-Sequence, $$\frac{\partial \ln P_{\text{Seq}}(y \mid q)}{\partial s_j} = P(p_j \mid q, y) - P_\eta(p_j \mid q),$$ the posterior of passage $p_j$ given the whole output minus its prior. For RAG-Token, $$\frac{\partial \ln P_{\text{Tok}}(y \mid q)}{\partial s_j} = \sum_{t=1}^{T} \big(\rho_t(j) - \pi_j\big),$$ the sum over tokens of the per-token posterior minus the prior.
Proof. The generator's probabilities do not depend on the scores. For RAG-Sequence, $$\frac{\partial P_{\text{Seq}}}{\partial s_j} = \sum_i G_i\, \pi_i (\delta_{ij} - \pi_j) = \pi_j G_j - \pi_j \sum_i \pi_i G_i = \pi_j G_j - \pi_j P_{\text{Seq}},$$ and dividing by $P_{\text{Seq}}$ gives $\pi_j G_j / P_{\text{Seq}} - \pi_j$, where the first term is the posterior $P(p_j \mid q, y)$ by Bayes' rule. For RAG-Token, $\ln P_{\text{Tok}} = \sum_t \ln \sum_i \pi_i a_t(i)$; the same computation applied to each term, with $a_t(i)$ in place of $G_i$, gives $\rho_t(j) - \pi_j$, and the derivative of the sum is the sum of the derivatives. $\blacksquare$
The gradient with respect to the query encoder's parameters follows by the chain rule, through $s(q, p_j) = E_Q(q) \cdot E_P(p_j)$: each retrieved passage's vector pulls the query vector toward itself with weight posterior minus prior. Passages that explain the output better than the retriever expected are pulled closer; the others are pushed away.
REALM writes the same gradient in another form. With $f(x, z)$ its retrieval score and $p(z \mid x)$ its retrieval distribution, $\nabla \log p(y \mid x) = \sum_z r(z)\, \nabla f(x, z)$ with $$r(z) = \left[\frac{p(y \mid z, x)}{p(y \mid x)} - 1\right] p(z \mid x),$$ and "document $z$ receives a positive update whenever it performs better than expected" (Guu et al., 2020, §3.3). Multiplying out, $r(z) = p(y \mid z, x)\, p(z \mid x)/p(y \mid x) - p(z \mid x)$, which is posterior minus prior: Proposition 5 for RAG-Sequence. The script checks both closed forms against finite differences and checks that REALM's $r$ equals the posterior minus the prior on the example.
In the running example the posterior of $p_2$ after the whole output is $9/13$ and its prior $1/4$, so RAG-Sequence's gradient on $s(q, p_2)$ is $9/13 - 1/4 = 23/52 = 0.442$, and on $s(q, p_1)$ it is $-0.442$. RAG-Token's is $(2/3 - 1/4) + (3/11 - 1/4) + (1/4 - 1/4) = 29/66 = 0.439$. Both raise the score of the passage that names the novel, as they should, though the training example never said which passage that was.
The generator's gradient. The same computation with respect to the generator's parameters gives, for RAG-Sequence, $$\frac{\partial \ln P_{\text{Seq}}}{\partial \theta} = \sum_i P(p_i \mid q, y)\, \frac{\partial \ln G_i}{\partial \theta},$$ since $\partial P_{\text{Seq}}/\partial \theta = \sum_i \pi_i G_i\, \partial \ln G_i/\partial \theta$. The generator is trained to produce the output from each passage, in proportion to that passage's posterior. This is the weighting of the E-step of the EM algorithm, applied here inside gradient descent: the same posterior that probabilistic latent semantic analysis computes for its hidden topics.
Why retrieval collapses
Proposition 6 (collapse). If the generator ignores its passage, so that $a_t(i)$ does not depend on $i$ for the training outputs, then the retriever's gradient is zero for both models.
Proof. If $a_t(i) = a_t$ for all $i$, the posterior $P(p_j \mid q, y) = \pi_j \prod_t a_t / \sum_i \pi_i \prod_t a_t = \pi_j$, so the RAG-Sequence gradient is $\pi_j - \pi_j = 0$; likewise $\rho_t(j) = \pi_j$ for every $t$, so the RAG-Token gradient is zero. $\blacksquare$
Proposition 6 makes a self-reinforcing loop possible. A generator that finds its passages unhelpful may learn to ignore them, and once it does, the retriever receives no training signal from which to start retrieving better ones. Guu et al. (2020, §3.4) describe this loop as a "cold-start problem" and avoid it by warm-starting REALM's retriever with the inverse cloze task, in which a sentence is the query and its surrounding passage the target. Lewis et al. (2020, Appendix H) report a collapse for story generation, in the other order: the retriever "would 'collapse' and learn to retrieve the same documents regardless of the input", and then "the generator would learn to ignore the documents, and the RAG model would perform equivalently to BART". They suggest causes that Proposition 6 does not address: "a less-explicit requirement for factual knowledge in some tasks, or the longer target sequences, which could result in less informative gradients for the retriever". What the proposition explains is why, in such a state, the retriever receives no gradient: only the generator's own training could end it, and nothing in the retriever's update pushes it to. RAG starts from DPR's retriever, already trained with question-passage pairs on Natural Questions and TriviaQA (§2.2), and the paper initializes from it throughout; that was enough for the question-answering tasks but, by Appendix H, did not prevent a collapse in preliminary experiments on story generation.
A second limit follows from the top-$k$ approximation. A passage outside the top $k$ is not in the sum, so it receives no gradient at all. The retriever can learn only about passages it already ranks near the top, which is another reason the starting retriever matters.
Decoding
At test time the model must produce the most probable output, or a good approximation of it (§2.5). RAG-Token is an ordinary autoregressive model with next-token distribution $\sum_i \pi_i a_t(i)$, so a standard beam search applies. RAG-Sequence does not factor into per-token probabilities with fixed weights (Proposition 2 shows its weights change with the prefix), so the paper runs a separate beam search for each passage and then scores the union $Y$ of the hypotheses found.
- Thorough Decoding computes $P_{\text{Seq}}(y \mid q) = \sum_i \pi_i G_i(y)$ exactly for each $y \in Y$, running "an additional forward pass for each document $z$ for which $y$ does not appear in the beam".
- Fast Decoding approximates $G_i(y) \approx 0$ for every passage whose beam did not produce $y$, which saves those passes.
For question answering, where outputs are short, the paper used Thorough Decoding, with greedy decoding (a beam of one) "as we did not find beam search improved results" (Appendix A). Fast Decoding can choose a different answer, as worked problem 4 shows.
REALM: the same model for pre-training
REALM (Guu et al., 2020) wrote the same marginal likelihood a few months before RAG: $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(y \mid z, x)\, p(z \mid x),$$ with retrieval $p(z \mid x) \propto \exp f(x, z)$ and $f(x, z)$ the inner product of two BERT embeddings projected to a lower dimension (§3.1 to §3.2). Its purpose was different. REALM used the model during masked-language-model pre-training: $x$ is a sentence with a span masked out, $y$ the missing span, and the retriever is trained to find passages that help predict it. The knowledge-augmented encoder that reads $x$ together with a passage $z$ is a BERT network, which predicts each masked token during pre-training and, when fine-tuned for question answering, scores the answer's start and end in $z$, an extractive reader in the sense of the previous article.
Several of REALM's devices address problems this article has derived.
- A stale index. Because REALM trains the passage encoder, the stored passage vectors go out of date after every update. REALM re-embeds and re-indexes the whole collection asynchronously, in a second job, "every several hundred training steps", and recomputes $p(z \mid x)$ with fresh parameters for the top $k$ it retrieves (§3.3); refreshes came about every 500 steps (§4.5). A 30 times slower refresh cut exact match on Natural Questions from 38.2 to 28.7 (Table 2).
- Masking salient spans. REALM masks named entities and dates, such as "United Kingdom" or "July 1969", rather than random tokens, to focus on predictions that need world knowledge (§3.4). With random token masks the ablation reached 32.3 and with random span masks 35.3, against 38.2 (Table 2).
- A null document. An empty document is added to the top $k$, "allowing appropriate credit to be assigned to a consistent sink when no retrieval is necessary" (§3.4).
- Trivial retrievals. When the pre-training corpus is the retrieval corpus, the passage containing the masked sentence would give the answer away; it is excluded (§3.4).
- Warm starts, as above: the inverse cloze task for the retriever and BERT-base for the encoder.
The paper's Table 3 shows the latent variable at work on one masked word. For the sentence "An equilateral triangle is easily constructed using a straightedge and compass, because 3 is a ___ prime" BERT gives the correct word, "Fermat", a probability of $1.1 \times 10^{-14}$. Given a retrieved passage saying "257 is ... a Fermat prime", REALM's encoder gives it 1.0, and the marginal over the top 8 retrieved documents is 0.129.
Pre-trained on Wikipedia or on news, REALM reached 39.2 and 40.4 exact match on NaturalQuestions-Open, against 33.3 for ORQA (Lee, Chang and Toutanova, 2019), which shares its fine-tuning setup, and 34.5 for the 11-billion-parameter closed-book T5 (Table 1). Its retriever's recall@5 before fine-tuning, the share of questions with the gold answer in the top five, was 38.5 against ORQA's 13.9 (Table 2).
The two papers differ mainly in the reader and in what is trained. REALM trains its retriever's passage encoder during pre-training and, for question answering, reads with an extractive span model; RAG starts from a supervised retriever, never retrains the passage encoder, and generates.
What the RAG paper reports
All numbers in this section are the paper's.
Open-domain question answering. Table 1 gives exact match on the test sets of four datasets (for TriviaQA, the open-domain test set and, after the slash, the TriviaQA Wiki test set used by the closed-book T5):
| Model | NQ | TriviaQA | WebQuestions | CuratedTREC |
|---|---|---|---|---|
| T5-11B, closed book | 34.5 | – / 50.1 | 37.4 | – |
| T5-11B + SSM, closed book | 36.6 | – / 60.5 | 44.7 | – |
| REALM | 40.4 | – | 40.7 | 46.8 |
| DPR | 41.5 | 57.9 | 41.1 | 50.6 |
| RAG-Token | 44.1 | 55.2 / 66.1 | 45.5 | 50.0 |
| RAG-Sequence | 44.5 | 56.8 / 68.0 | 45.2 | 52.2 |
The DPR row is an extractive system with a cross-encoder re-ranker. The paper reports RAG-Sequence with 50 retrieved documents and RAG-Token with 15 at test time (Appendix A), and notes that "RAG can generate correct answers even when the correct answer is not in any retrieved document, achieving 11.8% accuracy in such cases for NQ" (§4.1), which escapes the bound that Open-Domain Question Answering proves for extractive readers.
Number of documents. Figure 3 (left) shows that retrieving more documents at test time "monotonically improves Open-domain QA results for RAG-Sequence, but performance peaks for RAG-Token at 10 retrieved documents" (§4.5). Training with 5 or 10 documents made no significant difference.
Ablations on the development sets (Table 6, exact match on Natural Questions): replacing the trained retriever by a fixed BM25 retriever, with BM25 scores as the logits of $P_\eta$, gave 29.7 for RAG-Token and 31.8 for RAG-Sequence; freezing the retriever at DPR's weights gave 37.8 and 41.2; training it end to end gave 43.5 and 44.0. On FEVER, a fact-verification task whose claims are "heavily entity-centric", BM25 was best.
Generation. On Jeopardy question generation, RAG-Token scored above RAG-Sequence and BART on the automatic measures (Table 2). In a human evaluation of 452 pairs of clues from BART and RAG-Token, the assessors judged RAG more factual in 42.7% of pairs and BART more factual in 7.1% (Table 4). The ratio of distinct to total trigrams, a measure of diversity, was 46.8% for RAG-Token, 53.8% for RAG-Sequence and 32.4% for BART on Jeopardy (Table 5).
Sizes. The model holds the two BERT-base encoders of DPR, 110 million parameters each, and BART-large's 406 million. The paper totals these as "626M trainable parameters" while noting that it does not train the passage encoder (Appendix G), so 516 million parameters are trained. For the index it gives "21M 728 dimensional vectors, consisting of 15.3B values". DPR's BERT-base vectors have 768 dimensions (Karpukhin et al., 2020, §3.1), which with the 21,015,324 passages of DPR's collection gives 16.1 billion values; 15.3 billion is the product with 728 as printed. Training used eight 32GB GPUs; the index was held on the CPU in about 100 GB of memory, reduced to 36 GB with FAISS's compression (Appendix C).
What is proved, what is cited, what the script shows
Proved here. That both likelihoods are probability distributions (Proposition 1); RAG-Sequence written token by token, with posterior weights against RAG-Token's prior weights (Proposition 2); sufficient conditions for the two to agree (Proposition 3); the two-token identity $P_{\text{Tok}} - P_{\text{Seq}} = -\operatorname{Cov}_\pi(a_1, a_2)$ (Proposition 4) and its three-token analogue with a third central moment; that RAG-Sequence gives the higher probability whenever the passages can be ordered so that every token's support is nondecreasing (Proposition 4b); the retriever's gradient as posterior minus prior for RAG-Sequence and as a sum of per-token posteriors minus the prior for RAG-Token, and its equality with REALM's $r(z)$ (Proposition 5); the generator's gradient as a posterior-weighted sum; and that a generator which ignores its passages gives the retriever no gradient (Proposition 6).
Cited. The models' architecture, training and decoding procedures and every reported result of Lewis et al. (2020) and Guu et al. (2020), with the table, figure or section named; ORQA's comparison numbers through REALM's Table 1; DPR's dimension from Karpukhin et al. (2020).
Computed by the script. The running example, the gradients (also against finite differences), the two-token identity on the second example and on 2,000 random cases, the three-token identity on 2,000 more, the count of 10,000 random three-token cases, Proposition 4b on 20,000 random cases, the collapse case, the decoding problem and the arithmetic on the reported sizes. The example's probabilities are invented to show the mechanics, and the random counts describe the random cases drawn, not any trained model.
Worked problems
The unit has no course assignments; these problems use the running example and small constructed cases, and the script recomputes each number.
Problem 1 (the posterior, token by token). In the running example, compute RAG-Sequence's posterior over the two passages after each token, and RAG-Token's per-token posteriors. Explain why the RAG-Sequence posterior stops moving after the second token.
Before any token the posterior is the prior, $(3/4, 1/4)$. After "Sun" the unnormalized weights are $\tfrac34 \cdot \tfrac1{10} = 0.075$ and $\tfrac14 \cdot \tfrac35 = 0.15$, so the posterior is $(1/3, 2/3)$. After "Also" they are $0.075 \cdot \tfrac45 = 0.06$ and $0.15 \cdot \tfrac9{10} = 0.135$, giving $(4/13, 9/13)$, that is $(0.308, 0.692)$. After "Rises" both weights are multiplied by $9/10$, so their ratio, and the posterior, do not change: a token that both passages predict equally carries no evidence about which passage is being used. RAG-Token's per-token posteriors of $p_2$ are $0.15/0.225 = 2/3$, $0.225/0.825 = 3/11$ and $0.225/0.9 = 1/4$: each is computed from the prior and that token alone, so the third returns exactly to the prior.
Problem 2 (the sign of the difference). Two passages have equal prior, $\pi = (1/2, 1/2)$, and the target has two tokens. (a) $p_1$ gives the tokens probabilities $1/10$ and $4/5$, $p_2$ gives $4/5$ and $1/10$. (b) $p_1$ gives $4/5$ and $4/5$, $p_2$ gives $1/10$ and $1/10$. Compute both likelihoods and check Proposition 4.
(a) $P_{\text{Seq}} = \tfrac12 (0.08) + \tfrac12 (0.08) = 0.08$; $P_{\text{Tok}} = (0.45)(0.45) = 0.2025$. The covariance is $\mathbb{E}[a_1 a_2] - \mathbb{E}[a_1]\mathbb{E}[a_2] = 0.08 - 0.2025 = -0.1225$, and indeed $P_{\text{Tok}} - P_{\text{Seq}} = 0.1225$. Each passage supports one token, so the model that can switch passages wins. (b) $P_{\text{Seq}} = \tfrac12(0.64) + \tfrac12(0.01) = 0.325$; $P_{\text{Tok}} = 0.45^2 = 0.2025$; the covariance is $0.325 - 0.2025 = +0.1225$, and RAG-Sequence wins by that amount. The marginal token probabilities are the same in (a) and (b); only their pairing across passages differs, and that pairing is all that decides the comparison.
Problem 3 (collapse in numbers). In the running example, replace the generator's table by one that ignores the passage: both rows $(1/5, 7/10, 9/10)$. What are the likelihoods and the retriever's gradients?
Both likelihoods equal $\tfrac15 \cdot \tfrac7{10} \cdot \tfrac9{10} = 0.126$, by Proposition 3. The posterior after the whole output is $\pi_i \cdot 0.126 / 0.126 = \pi_i$, so the RAG-Sequence gradient on each score is $\pi_i - \pi_i = 0$; each per-token posterior is also $\pi_i$, so the RAG-Token gradient is 0 too. Training on this example leaves the retriever exactly where it was, whatever passages it retrieved: Proposition 6 on numbers.
Problem 4 (Thorough and Fast decoding). A RAG-Sequence model has two passages with prior $(7/10, 3/10)$, a vocabulary of two tokens $u$ and $v$, and outputs of two tokens. Reading $p_1$, the generator gives the first token $u$ with probability $11/20$; after $u$ the second is $u$ with $3/5$, after $v$ it is $v$ with $19/20$. Reading $p_2$, the first token is $v$ with $9/10$; after $v$ the second is $v$ with $5/9$, after $u$ either token with $1/2$. With a beam of one per passage, which output does each decoding procedure return, and which output is most probable?
Greedy decoding of $p_1$ picks $u$ (11/20) then $u$ (3/5): the beam holds $uu$, with $G_1(uu) = 0.33$, although $G_1(vv) = \tfrac9{20} \cdot \tfrac{19}{20} = 0.4275$ is larger, which a beam of one cannot see. Greedy decoding of $p_2$ gives $vv$, with $G_2(vv) = 0.5$. The candidates are $uu$ and $vv$.
Fast Decoding scores each candidate only with the passage that produced it: $uu$ gets $0.7 \times 0.33 = 0.231$ and $vv$ gets $0.3 \times 0.5 = 0.150$, so it returns $uu$. Thorough Decoding computes the full sums, with $G_2(uu) = \tfrac1{10} \cdot \tfrac12 = 0.05$: $uu$ gets $0.231 + 0.3 \times 0.05 = 0.246$ and $vv$ gets $0.7 \times 0.4275 + 0.150 = 0.449$, so it returns $vv$. The exact distribution over the four outputs is $uu$ 0.246, $uv$ 0.169, $vu$ 0.136, $vv$ 0.449, so Thorough Decoding found the most probable output. Fast Decoding lost because $vv$'s largest contribution came from a passage whose beam missed it.
Problem 5 (the size of the non-parametric memory). How many numbers does RAG's index hold, and how many bytes at 32-bit, 16-bit and 8-bit precision?
The collection has 21,015,324 passages (Karpukhin et al., 2020, §4.1), each a 768-dimensional vector: $21{,}015{,}324 \times 768 = 16{,}139{,}768{,}832$, about 16.1 billion numbers. At 4 bytes each that is about 64.6 GB, at 2 bytes 32.3 GB, and at 1 byte 16.1 GB. The paper reports about 100 GB of CPU memory for the index of all of Wikipedia, and 36 GB after FAISS's compression tools, without breaking either figure down or giving the compression settings (Appendix C). For comparison, the model's 626 million parameters are 3.9% of the index's 16.1 billion numbers, and the 516 million it trains are 3.2%. Most of the numbers that hold the model's knowledge of Wikipedia are in the index, which can be inspected passage by passage and replaced without training, as the world-leaders experiment showed.
What we have learned
RAG and REALM treat the retrieved passage as a hidden variable and maximize the probability of the output summed over the top $k$ passages. RAG-Sequence sums once over complete outputs; RAG-Token sums at every token. Written one token at a time, RAG-Sequence weights the passages by a posterior that sharpens as the output is generated, while RAG-Token weights them by the retriever's prior at every step. For two tokens the difference between their likelihoods is minus the covariance of the passages' support for the two tokens; for any length RAG-Sequence gives the higher probability when the passages agree in their support for every token, and otherwise either can. The training signal for the retriever is posterior minus prior: a passage is pulled toward the query when it explains the output better than the retriever expected. That signal vanishes when the generator ignores its passages, so while the generator ignores its passages the retriever has no signal to recover from; REALM warm-starts its retriever for that reason, and RAG starts from DPR's. The published results show the gains on open-domain question answering and the knowledge that can be updated by replacing the index.
Both models read each passage separately and combine their outputs, so a generator never sees two passages at once. Fusion-in-Decoder and Reading Many Passages lets the decoder attend to all of them together, and counts what that costs.
References
- Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.-W. (2020). Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, PMLR 119, 3929–3938. arXiv:2002.08909. record
- Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP 2020, 6769–6781. https://doi.org/10.18653/v1/2020.emnlp-main.550. arXiv:2004.04906. record · 15 pages
- Lee, K., Chang, M.-W., and Toutanova, K. (2019). Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 6086–6096. https://doi.org/10.18653/v1/P19-1612. arXiv:1906.00300. record · 4 pages
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33, 9459–9474. arXiv:2005.11401. record · 6 pages
- Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., and Shoham, Y. (2023). In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11, 1316–1331. https://doi.org/10.1162/tacl_a_00605. arXiv:2302.00083. record · 2 pages