Assembling the Context

What happens between retrieval and reading when the reader is a frozen language model. The in-context design and its token cost; why more passages raise the chance that the answer is present while the reader's accuracy can fall, written as an exact decomposition; the position effect of Liu et al. and a recorded replication with Claude Opus 5.5 and Claude Haiku 5.5 on their twenty-passage questions, tested pair by pair; the ordering rule that follows from a slot-weight model, proved by an exchange argument; and redundancy and the token budget, with top-k selection proved optimal for equal passages and a half-optimal rule proved for unequal ones. Five worked problems apply each result to the recorded runs.

Twenty passages are placed in front of a language model with the question who was in the "Don't Worry, Be Happy" video? Nineteen of them came from a retriever and do not contain the answer; eleven come from the article on a later song of the same name by Guy Sebastian. One passage, from the article on Bobby McFerrin's 1988 song, says that its comedic original video stars McFerrin, Robin Williams and Bill Irwin. When that passage is the first of the twenty, Claude Haiku 5.5 names McFerrin, Williams and Irwin. When the same passage is the tenth, the same model answers "Guy Sebastian". Nothing was added or removed; only the order changed. How should retrieved passages be placed in a model's input: how many, in what order, and within what budget of tokens?

Context assembly is the step that turns a ranked list of retrieved passages into the text a language model reads. It decides which of the $k$ retrieved passages $p_1, \dots, p_k$ to include, how many, in what order and format, and how to fit them, together with the question $q$ and any instructions, into a limited number of tokens. The retriever's job ends with the ranked list; the reader's begins with the assembled context. This article is about what lies between.

Here is the route. First we describe the in-context design, in which retrieved text is placed in front of a model that was never trained with the retriever, and what it costs. Then we take the number of passages, and write the reader's accuracy as an exact sum of two parts, one that rises with $k$ and one that need not. Next comes order: the position effect reported by Liu et al. (2024), and our own recorded replication on their questions with two current models, followed by the ordering rule that a simple model of position implies, with its proof. Finally we treat redundancy and the token budget, and prove when choosing the top passages is optimal. The article assumes Open-Domain Question Answering for the task and its measures, and links to Result Diversification for redundancy. The numbers are computed by tools/experiments/rag/assembling-the-context.py from the runs recorded by context_runs.py.

The in-context design

A retrieval-augmented reader can be built in two ways. In the first, the retriever and the reader are trained together, so that the reader learns to use passages and the retriever learns which passages the reader can use; the models of RAG-Sequence and RAG-Token and Fusion-in-Decoder are of this kind. In the second, the reader is a language model trained for general use and left unchanged, and the retrieved passages are simply written into its input.

Ram et al. (2023) named the second design In-Context RALM (retrieval-augmented language modeling). For a language model that predicts each token from the ones before it, they write it (their Equation 2) as

$$P(x_1, \dots, x_n) = \prod_{i=1}^{n} P_\theta\big(x_i \mid [R_{\mathcal{C}}(x_{<i});\, x_{<i}]\big),$$

where $R_{\mathcal{C}}(x_{<i})$ is the text retrieved from the corpus $\mathcal{C}$ for the prefix $x_{<i}$, $[a; b]$ is the concatenation of two strings, and $\theta$ are the model's parameters, which are not changed. In the notation of this unit, a question-answering reader of this kind produces its answer $y = (y_1, \dots, y_T)$ from $P_\theta(y_t \mid c, y_{<t})$, with the context

$$c = [\text{instruction};\ p_1;\ p_2;\ \dots;\ p_k;\ q].$$

The design asks nothing of the model's training, which is its attraction. Ram et al. report that prepending retrieved documents with off-the-shelf retrievers gave language-modeling gains that they equate with increasing the model's size by two to three times across the corpora they examined, and that a 345-million-parameter GPT-2 with a BM25 retriever outperformed a 762-million-parameter GPT-2 without retrieval. Every choice the training would otherwise have made is now made by whoever assembles $c$.

Shi et al. (2024) proposed a variant, REPLUG, that does not concatenate the passages. Each passage is prepended to the input separately, the model is run once per passage, and the next-token distributions are averaged with weights from the retriever's scores:

$$P(y_t \mid q, y_{<t}) = \sum_{j=1}^{k} \lambda_j\, P_\theta\big(y_t \mid [p_j; q], y_{<t}\big), \qquad \lambda_j = \frac{e^{s(q, p_j)}}{\sum_{i=1}^{k} e^{s(q, p_i)}}.$$

This is the per-token mixture of RAG-Token with a frozen generator. Shi et al. motivate it by the context window: concatenation is limited by how many documents fit. They report that ensembling random documents instead of retrieved ones was worse, so the gain is not an effect of ensembling as such. The rest of this article concerns the concatenated form, the one available when a model's interface returns text rather than next-token probabilities, as with the models used below.

What the context costs. A model reached through such an interface is billed by the token, for input and output separately. In the runs of this article, a question alone made a median prompt of 52 tokens for Opus 5.5; the answering passage alone, 284; BM25's top five passages, 1,203.5; and twenty passages (the five position conditions together), 4,576.5 (a median of an even number of prompts can end in a half). The input bill grows linearly with $k$. Liu et al. (2024) note that the Transformer's memory and compute grow quadratically in sequence length, and that Transformer language models were for that reason often trained with small context windows. The token count is the first constraint on assembly. Whether more tokens buy more accuracy is the second, and the subject of the next section.

How many passages: an exact decomposition

More passages raise the chance that the answer is somewhere in the context. Whether the reader then finds it is a separate matter. The two can be separated exactly.

Fix a retriever and a reader, and draw a question at random from a set. Let $A_k$ be the event that at least one of the top $k$ passages contains a gold answer string; its probability is the retriever's top-$k$ accuracy, $R(k) = P(A_k)$, as open-domain question answering defines it. Let $C_k$ be the event that the reader, given those $k$ passages, answers correctly. Write

$$a_1(k) = P(C_k \mid A_k), \qquad a_0(k) = P(C_k \mid \text{not } A_k)$$

for the reader's accuracy when the answer is available and when it is not.

Proposition 1 (decomposition). $P(C_k) = R(k)\, a_1(k) + \big(1 - R(k)\big)\, a_0(k)$, and $R(k)$ is nondecreasing in $k$.

Proof. $A_k$ and its complement partition the questions, so the law of total probability gives $P(C_k) = P(C_k \mid A_k)P(A_k) + P(C_k \mid \text{not } A_k)P(\text{not } A_k)$, which is the first claim. For the second, the top $k$ passages are a subset of the top $k+1$, so a question whose top $k$ contain an answer string has top $k+1$ that contain it too: $A_k \subseteq A_{k+1}$, and $P(A_k) \le P(A_{k+1})$. $\blacksquare$

The proposition says what can and cannot be promised. Raising $k$ never lowers $R(k)$, the retriever's contribution. It can lower $a_1(k)$, because the reader must find the answer among more passages, and it can raise or lower $a_0(k)$, the accuracy when the answer is absent, which comes from the model's own memory and from how it reacts to passages that do not help. Accuracy rises with $k$ only if the gain in $R$ outweighs the losses in $a_1$ and $a_0$.

Example (Opus 5.5 with BM25). For the 150 questions of the Opus runs described below, BM25 over a 50,769-passage corpus put an answer string in the top 1, 5 and 20 passages for 25, 71 and 122 questions. Opus 5.5 answered 94, 94 and 114 correctly. With five passages it was right on 60 of the 71 questions whose answer was available, $a_1(5) = 0.845$, and on 34 of the 79 whose answer was not, $a_0(5) = 0.430$. The identity checks: $\tfrac{71}{150} \times \tfrac{60}{71} + \tfrac{79}{150} \times \tfrac{34}{79} = \tfrac{94}{150}$. Without any passage the same model answered 104 of the 150 correctly. Five retrieved passages therefore gave a lower accuracy than none, 0.627 against 0.693, although the difference is not decisive (McNemar's exact test, $p = 0.087$; 9 questions right only with the passages, 19 right only without). The reason is visible in $a_0$, and it is not mainly reticence. The prompt asks for an answer "using only the provided search results". Opus answered 19 questions closed-book but missed them with the five passages, and for 18 of them the passages lacked the answer. A reading of those 18 responses (by an instance of Claude Opus 5.5, the writer of this article) finds 3 that say the passages do not give the answer, 2 that give an adjacent answer and say the sources lack the one asked for, 11 confident wrong answers taken from or prompted by the passages, 1 that answers the question as literally worded (the 1970 film of that title, where the annotation means the 2005 film), and 1 that gives the right year without the annotated date, which the reading counts as correct. When the five passages lack the answer, they mislead this reader more often than they silence it; Evaluating Retrieval-Augmented Systems gives the full accounting.

The corpus matters for every number in this example. It is not a sample of Wikipedia: it is every passage of Liu et al.'s 2,655 twenty-passage examples, described below, 50,769 passages in all. Each question's answering paragraph is in it by construction, together with the 19 passages that a dense retriever ranked highest for that question among those lacking its answer, and the rest are the passages of the other questions. The retrieval figures here are properties of this deliberately adversarial corpus, not of BM25 on Wikipedia. Twenty passages put the answer within reach for 122 questions and gave 114 correct answers, 0.760.

Liu et al. (2024) ran the same kind of study with Contriever retrieving from Wikipedia for NaturalQuestions-Open and report that reader performance saturates long before retriever recall does: using 50 documents instead of 20 improved accuracy by about 1.5 points for GPT-3.5-Turbo and about 1 point for Claude-1.3, while making the input much longer. They suggest reranking the retrieved passages and truncating the ranked list as remedies. Both are decisions of context assembly.

In what order: the position effect

The opening example is a case of a broader pattern. Liu et al. (2024) measured it in a controlled form, and we repeated their experiment on two current models.

The experiment of Liu et al.

Their task, multi-document question answering, gives the model a question and $k$ passages of which exactly one contains the answer. The questions are the 2,655 of NaturalQuestions-Open (Lee et al., 2019; Kwiatkowski et al., 2019) whose annotated long answer is a paragraph. The answering passage is that paragraph, cut to a chunk of at most 100 tokens. The other $k - 1$ passages are the highest-ranked Wikipedia passages that a Contriever retriever fine-tuned on MS MARCO returned for the question and that contain none of the annotated answers, presented in order of decreasing relevance. Moving the answering passage to a chosen position changes nothing else, and adding distractors changes the length without changing the answer. A response counts as correct if any annotated answer appears in it. The models were given a standard prompt and decoded greedily.

Liu et al. report a U-shaped curve: performance is often highest when the answering passage comes first or last and lowest when it is in the middle, which they call primacy and recency bias. For GPT-3.5-Turbo, accuracy with the passage in the middle of 20 or 30 documents fell below its closed-book accuracy of 56.1 percent, the accuracy with no documents at all (their Table 1 gives the closed-book and oracle accuracies, the oracle being the answering passage alone; for Claude-1.3 they were 48.3 and 76.1 percent). Extended-context versions of the same models behaved almost identically to the originals when both could hold the input. Their further analyses found that encoder-decoder models were relatively robust while the input stayed within their training length (1.9 points between best and worst position for Flan-UL2), that placing the question both before and after the documents barely changed the question-answering trends, and that a base model without instruction tuning already showed the U shape.

A recorded replication on current models

The observations below are of one dated run, recorded on 18 November 2026 with its prompts and every response. They describe these two models on these questions and are not general findings. Anthropic's documentation says to omit temperature, top_p and top_k for both models (for Opus 5.5 they return an error), and its Opus 5.5 migration guide notes that a temperature of 0 never guaranteed identical outputs on earlier models (Anthropic, 2026a, 2026b). The run therefore cannot be repeated exactly; its records are the result, and a new run would be a new observation.

Design. We used Liu et al.'s released data for 20 passages, which places the answering passage at positions 1, 5, 10, 15 and 20 (their index 0, 4, 9, 14 and 19). Our script rebuilds each arrangement from their position-1 file and checks it against their published position-10 file, identical for all 300 questions we drew. We kept only questions whose answering passage contains an annotated answer under their scorer's normalization and whose 19 distractors contain none, 2,345 of the 2,655, because a distractor that contains an answer string makes the position of "the" answering passage ambiguous. From these we drew 300 questions with seed 20261118. The questions and annotated answers are NQ-Open's (released by Google under CC BY-SA 3.0), and the passages are Wikipedia text (CC BY-SA) as distributed with Liu et al.'s code (MIT licence); the questions and passage text quoted in this article and the two that follow come from there. Claude Haiku 5.5 answered all 300 and Claude Opus 5.5 the first 150 drawn, in seven conditions each: closed-book, oracle, and the five positions. The prompt was Liu et al.'s, verbatim: "Write a high-quality answer for the given question using only the provided search results (some of which might be irrelevant)", then the documents in their format, Document [n](Title: t) text, then the question. We added a system prompt asking for "a short phrase or a single sentence, with no explanation", because both models otherwise write paragraphs and Liu et al.'s scorer reads only the first line of a response. Thinking is always on for Opus 5.5 (Anthropic, 2026c); we set its effort to low, as for Haiku. Correctness is Liu et al.'s: an annotated answer, after lowercasing and removing punctuation and articles, appears in the first line (we first remove leading blank lines, which their script does not).

Results.

Condition Opus 5.5 (150 questions) Haiku 5.5 (300 questions)
Closed-book 104 (0.693) 163 (0.543)
Answer at position 1 of 20 115 (0.767) 228 (0.760)
Position 5 120 (0.800) 213 (0.710)
Position 10 118 (0.787) 212 (0.707)
Position 15 117 (0.780) 209 (0.697)
Position 20 112 (0.747) 215 (0.717)
Oracle (answering passage alone) 133 (0.887) 269 (0.897)
Accuracy by the position of the answering passage among 20, for Opus 5.5 (150 questions) and Haiku 5.5 (300), with each model's closed-book and oracle accuracy as dotted lines. Notice that Haiku's curve drops after position 1 and stays flat, a primacy effect without the recency end of a U; Opus's curve varies by a few questions with no pattern the paired tests can separate from chance; and both stay well below their oracle lines, so the 19 distractors cost accuracy wherever the answer sits.
Accuracy by the position of the answering passage among 20, for Opus 5.5 (150 questions) and Haiku 5.5 (300), with each model's closed-book and oracle accuracy as dotted lines. Notice that Haiku's curve drops after position 1 and stays flat, a primacy effect without the recency end of a U; Opus's curve varies by a few questions with no pattern the paired tests can separate from chance; and both stay well below their oracle lines, so the 19 distractors cost accuracy wherever the answer sits.

Each comparison is paired: the same questions are answered in both conditions, so the test is McNemar's exact test, which uses only the questions answered correctly in one condition and not the other. Under the hypothesis that the two conditions are equally accurate, each such discordant question falls either way with probability one half, and the two-sided $p$-value is twice the binomial tail. We report the two comparisons that Liu et al.'s U shape names in advance, start against middle and end against middle:

Comparison Opus: right only in first, only in second, $p$ Haiku: same
Position 1 against 10 2, 5, $p = 0.453$ 24, 8, $p = 0.007$
Position 20 against 10 2, 8, $p = 0.109$ 12, 9, $p = 0.664$
Position 1 against closed-book 15, 4, $p = 0.019$ 81, 16, $p < 0.001$
Oracle against position 1 21, 3, $p < 0.001$ 43, 2, $p < 0.001$

Three observations follow, each about this run.

  • Haiku 5.5 shows primacy without recency. With the answer first it was right on 228 questions; anywhere else, on 209 to 215. Position 1 against position 10, the start-against-middle contrast that the U shape defines, is 24 questions to 8. The largest of Haiku's ten pairwise differences, chosen after looking, is position 1 against position 15, 26 questions to 7 ($p = 0.001$; 0.013 after a Bonferroni correction for ten comparisons). The end of the context gives no advantage over the middle (12 to 9). The curve is an L, not a U.
  • Opus 5.5 shows no position effect we can distinguish from chance. Its five accuracies range from 112 to 120, and neither named contrast is near significance. The largest difference, position 5 against position 20, is 8 questions to 0 ($p = 0.008$), but that pair was chosen after looking, from ten possible pairs; with a Bonferroni correction for ten comparisons it becomes $p = 0.078$.
  • Distractors cost both models accuracy, and retrieval still helped. Unlike GPT-3.5-Turbo in Liu et al.'s study, neither model fell to its closed-book accuracy at any position; Haiku's worst position, 209, is far above its closed-book 163. But both lost accuracy when 19 distractors were added, wherever the answer sat: Opus answered 133 questions with the answering passage alone and 112 to 120 with it among twenty, Haiku 269 and 209 to 228.

The substring rule misses correct answers worded differently from the annotations, and Evaluating Retrieval-Augmented Systems measures how often. Scored instead by a model judge (Haiku 5.5 comparing each response with the annotated answers, the judge of that article), Opus's five positions give 128, 131, 132, 131 and 129 correct answers, and Haiku's 259, 243, 247, 244 and 251. Opus stays flat. Haiku's first position stays highest, and its last position moves closer to it than under the substring rule.

The opening example is one of 24 questions that Haiku answered with the answer first and missed with it tenth. Opus answered that question correctly in all seven conditions.

An ordering rule, and when it applies

If position matters, the passages the retriever trusts most should go where the reader looks hardest. A simple model makes that precise, and an exchange argument proves the rule.

The model. Suppose exactly one of the $k$ passages carries the answer, and the retriever (or a re-ranker) assigns passage $j$ a probability $r_j$ of being that passage, with $\sum_j r_j = 1$. Suppose the reader answers correctly with probability $w_i$ when the answering passage sits in slot $i$, whatever the passage is. An ordering is a permutation $\sigma$ that puts passage $\sigma(i)$ in slot $i$, and the probability of a correct answer is

$$V(\sigma) = \sum_{i=1}^{k} w_i\, r_{\sigma(i)}.$$

Proposition 2 (sorted assignment). $V$ is maximized by putting the passage with the largest $r$ in the slot with the largest $w$, the second largest in the second best slot, and so on. Every ordering that pairs the two sorted lists this way achieves the maximum, and no other ordering exceeds it.

Proof. Take any ordering $\sigma$ and two slots $a$ and $b$ with $w_a > w_b$ but $r_{\sigma(a)} < r_{\sigma(b)}$: the better slot holds the less likely passage. Swap the two passages. Only the terms of slots $a$ and $b$ change, and

$$\big[w_a r_{\sigma(b)} + w_b r_{\sigma(a)}\big] - \big[w_a r_{\sigma(a)} + w_b r_{\sigma(b)}\big] = (w_a - w_b)\big(r_{\sigma(b)} - r_{\sigma(a)}\big) > 0.$$

So an ordering with such an inverted pair is not optimal. A maximum exists because there are finitely many orderings, so some ordering has no inverted pair. Take any ordering with no inverted pair and list its slots by decreasing $w$, breaking ties by decreasing $r$ of the passage placed there. Along this list the $r$-values never increase: two slots with different $w$ are not inverted, and two with equal $w$ were ordered by $r$. So $V(\sigma) = \sum_i w_{(i)} r_{(i)}$, where $w_{(1)} \ge w_{(2)} \ge \cdots$ and $r_{(1)} \ge r_{(2)} \ge \cdots$ are the two lists sorted, and this value is the same for every ordering without an inverted pair. Every optimal ordering is one of them, so they are exactly the optimal orderings, all with the same value. $\blacksquare$

This is the rearrangement inequality in the form context assembly needs. Its consequence depends entirely on the weights $w_i$. Where the reader favors the start, as Haiku did, the rule is to put the most probable passage first, which is the retriever's own order. If the slot weights fall in the order first, last, second, second-to-last and so on inward, as a U slightly higher at the start would make them (with a symmetric U the first and last slots tie, and either order of the top two passages is optimal), the second most probable passage belongs last, the third second, and so on inward, so that the least promising passages fill the middle. In general the rule is to sort the slots by their weights; a U-shaped curve need not alternate this way. Where the weights are equal, as Opus's were within the noise of this run, every order gives the same $V$, and order is not worth optimizing.

The model is an assumption and should be read as one. It treats the position effect as the same for every passage and question, and it ignores the interaction the opening example shows: the distractors there were damaging because they were about a different song of the same name, not merely because they came first. It also assumes exactly one useful passage. With several partly useful passages, the reader may need two of them together, and no slot weight describes that.

Redundancy and the token budget

The last decision is which passages to include at all, when the budget allows fewer than the retriever returned. Two complications arise: passages can repeat one another, and passages can differ in length.

Repetition. A context of five passages that say the same thing spends four slots for nothing. Ranking passages by their own relevance cannot see this, because each is scored alone. Result Diversification treats the problem in full: maximal marginal relevance penalizes a candidate for resembling the passages already chosen, and coverage-based methods choose passages to cover different aspects of a question. That article proves the properties of both and measures them on a five-passage context built from this site's pages. The twenty-passage example of the opening shows repetition doing harm of another kind: eleven passages from one wrong article surround the single passage from the right one, and a diversity criterion over article titles would have kept fewer of them.

Equal lengths. Suppose each passage $j$ has a value $v_j \ge 0$ for the answer, values add, and every passage has the same length, so a budget of $B$ tokens admits exactly $m$ passages.

Proposition 3. With additive values and equal lengths, the $m$ passages of highest value are an optimal selection.

Proof. Let $S$ be any selection of $m$ passages and $T$ the top $m$. If $S \ne T$, some passage $u \in S \setminus T$ and some $t \in T \setminus S$ exist, and $v_t \ge v_u$ because every passage of $T$ is at least as valuable as every passage outside it. Replacing $u$ by $t$ keeps $m$ passages and does not lower the total. Each replacement makes $S$ share one more passage with $T$, so at most $m$ replacements turn $S$ into $T$ without lowering the value. $\blacksquare$

This is a reason, beyond retrieval quality, for the fixed-length passages that Passages and Chunking discusses: with equal lengths the budget is a count, and the retriever's top $m$ is the right choice whenever values are additive.

Unequal lengths. If passage $j$ has length $\ell_j$ tokens, choosing a set of total length at most $B$ with the largest total value is the 0-1 knapsack problem. A natural rule takes passages in decreasing order of value per token, $v_j / \ell_j$, while they fit. It can fail badly. With $B = 1{,}000$, a passage $x$ of 1,000 tokens and value 0.5, and a passage $y$ of 100 tokens and value 0.1, the rule takes $y$ first (0.001 per token against 0.0005), after which $x$ no longer fits: it collects 0.1 where 0.5 was possible. A small repair guarantees half of the optimum.

Proposition 4. Let every passage have positive length, $\ell_j > 0$, and nonnegative value. Sort the passages by $\rho_j = v_j/\ell_j$, decreasing, and let the greedy rule take them in that order until the first passage $g$ that does not fit; call the value taken $G$. If no passage fails to fit, the greedy rule takes everything and is optimal. Otherwise, if every passage fits alone ($\ell_j \le B$), then $\max(G, v_g) \ge \tfrac12 \mathrm{OPT}$, where $\mathrm{OPT}$ is the best achievable value.

Proof. If the greedy rule takes every passage (total length at most $B$), taking all of them is feasible and, values being nonnegative, no selection is worth more. Otherwise consider the fractional problem, in which any fraction $x_j \in [0, 1]$ of a passage may be taken, with value $x_j v_j$ and length $x_j \ell_j$, subject to $\sum_j x_j \ell_j \le B$. Every whole selection is a fractional one, so the fractional optimum $F^*$ satisfies $F^* \ge \mathrm{OPT}$. Let $x^*$ take passages $1, \dots, g-1$ whole, the fraction $\theta = (B - \sum_{j<g}\ell_j)/\ell_g \in [0, 1)$ of passage $g$, and nothing after $g$; it uses the whole budget, $\sum_j x^*_j \ell_j = B$, and its value is $G + \theta v_g$. For any feasible $x$,

$$\sum_j (x^*_j - x_j)\, v_j = \sum_j (x^*_j - x_j)\,\ell_j\,\rho_j.$$

Compare each term with $\rho_g$. For $j < g$, $x^*_j - x_j \ge 0$ and $\rho_j \ge \rho_g$; for $j > g$, $x^*_j - x_j \le 0$ and $\rho_j \le \rho_g$; for $j = g$ the factor is $\rho_g$. In every case the term is at least $(x^*_j - x_j)\,\ell_j\,\rho_g$, and summing,

$$\sum_j (x^*_j - x_j)\, v_j \ \ge\ \rho_g\Big(B - \sum_j x_j \ell_j\Big) \ \ge\ 0.$$

So no feasible $x$ is worth more than $x^*$, and $F^* = G + \theta v_g$. Hence

$$\mathrm{OPT} \le F^* \le G + v_g \le 2\max(G, v_g).$$

The last step holds because the larger of two numbers is at least their average. Taking $g$ alone is feasible because it fits alone. $\blacksquare$

In the example the repaired rule compares $G = 0.1$ with $v_g = 0.5$ and takes $x$, which is optimal there. The bound is a guarantee, not a description of typical cases.

All three propositions assume that values add, and redundancy is exactly the failure of that assumption: two passages that say the same thing are worth less together than the sum of their values. That is where the methods of Result Diversification take over.

What is proved here, what is cited, and what the experiment shows

Proved here. The decomposition of reader accuracy into a retrieval term and two reader terms, and the monotonicity of top-$k$ accuracy (Proposition 1); the optimal ordering under the slot-weight model (Proposition 2); the optimality of the top $m$ passages for equal lengths and additive values (Proposition 3); the half-optimal guarantee of the repaired greedy rule for unequal lengths (Proposition 4).

Cited. The In-Context RALM formulation and its reported gains (Ram et al., 2023); REPLUG's ensemble and its comparison with random documents (Shi et al., 2024); the multi-document question-answering task, its data, the U-shaped curve, the closed-book and oracle accuracies and every other finding attributed to Liu et al. (2024); NaturalQuestions and its open version (Kwiatkowski et al., 2019; Lee et al., 2019); the models' sampling parameters, thinking and prices (Anthropic, 2026a, 2026b, 2026c, 2026d).

Observed in one recorded run. The accuracies, token counts and paired tests of Claude Opus 5.5 and Claude Haiku 5.5 on 150 and 300 of Liu et al.'s questions, recorded on 18 November 2026: primacy without recency for Haiku, no distinguishable position effect for Opus, a cost of 19 distractors for both, and for Opus five BM25 passages doing no better than none. The retrieval numbers are properties of the 50,769-passage corpus built from Liu et al.'s examples, not of BM25 on Wikipedia. Other models, other prompts, longer contexts and other questions may behave differently, and a rerun of the same prompts would not reproduce the responses exactly. The run cost $10.21 for Opus and $0.50 for Haiku at batch prices; every request and response is in tools/experiments/rag/data/runs/.

Worked problems

These problems are posed on the recorded runs of this article and on small constructed cases; the unit has no course assignments. The script recomputes every number.

Problem 1 (where the accuracy comes from). Haiku 5.5 with BM25's top 20 passages answered 203 of its 300 questions correctly. The answer was available in 248 of the 300 contexts, and the model was right on 191 of those. Find $R(20)$, $a_1(20)$ and $a_0(20)$, and check Proposition 1.

Solution. $R(20) = 248/300 = 0.827$. $a_1(20) = 191/248 = 0.770$. The 52 questions without an available answer contributed $203 - 191 = 12$ correct answers, so $a_0(20) = 12/52 = 0.231$. Then $R(20)\,a_1(20) + (1 - R(20))\,a_0(20) = \tfrac{248}{300}\cdot\tfrac{191}{248} + \tfrac{52}{300}\cdot\tfrac{12}{52} = \tfrac{191 + 12}{300} = \tfrac{203}{300} = 0.677$. The first term accounts for 191 of the 203 correct answers: with twenty passages, Haiku's accuracy is almost all retrieval. With five passages the answer was available for 144 questions, and the model was right on 120 of those and on 27 of the other 156, so $a_1(5) = 0.833$ and $a_0(5) = 0.173$. Going from 5 passages to 20 raised $R$ from 0.480 to 0.827 and lowered $a_1$ from 0.833 to 0.770: the reader found the answer slightly less often once it was present, and the gain came entirely from its being present more often.

Problem 2 (a paired test by hand). Haiku was right with the answer at position 1 and wrong with it at position 10 on 24 questions, and the reverse on 8. Compute McNemar's exact two-sided $p$-value.

Solution. There are $m = 32$ discordant questions. Under the hypothesis of equal accuracy, the number that favor position 1 is binomial with $m = 32$ and probability $\tfrac12$. The smaller count is 8, so

$$p = 2\sum_{x=0}^{8} \binom{32}{x} 2^{-32} = \frac{2 \times 15{,}033{,}173}{4{,}294{,}967{,}296} = 0.007.$$

The 268 questions answered alike in both positions do not enter the test; McNemar's test proves that they carry no information about the difference.

Problem 3 (ordering three passages). Three passages have probabilities $r = 0.6$, $0.3$ and $0.1$ of being the one that answers. Use Haiku's measured accuracies at positions 1, 10 and 20 of the run, $w = 228/300$, $212/300$ and $215/300$, as slot weights for three slots. Find the best and worst orderings under the model of Proposition 2, and the retriever's own order (most probable first).

Solution. Sorted, the slots rank first ($0.760$), last ($0.717$), middle ($0.707$). Proposition 2 pairs 0.6 with the first slot, 0.3 with the last and 0.1 with the middle: $V = 0.6 \times 0.760 + 0.1 \times 0.707 + 0.3 \times 0.717 = 0.7417$. The retriever's order puts 0.3 in the middle and 0.1 last: $V = 0.6 \times 0.760 + 0.3 \times 0.707 + 0.1 \times 0.717 = 0.7397$. The worst ordering reverses the pairing, 0.6 in the middle, 0.3 last and 0.1 first: $V = 0.1 \times 0.760 + 0.6 \times 0.707 + 0.3 \times 0.717 = 0.7150$. The best order gains 0.0020 over the retriever's and 0.0267 over the worst. (The products use the exact weights $228/300$, $212/300$ and $215/300$; the three-place values shown are rounded.) Under this model, putting the best passage first matters; arranging the rest hardly does, because Haiku's middle and end differ by one point.

Problem 4 (a budget with unequal passages). A budget of 1,200 tokens; four passages with lengths 600, 500, 400 and 300 tokens and values 0.42, 0.30, 0.32 and 0.15. Apply the ratio rule, the repaired rule of Proposition 4, and find the optimum.

Solution. The ratios are 0.42/600 = 0.000700, 0.30/500 = 0.000600, 0.32/400 = 0.000800 and 0.15/300 = 0.000500, so the order is the 400-token passage, the 600, the 500, the 300. The rule takes 400 and 600 (1,000 tokens, value 0.74); the 500-token passage does not fit, so $g$ is that passage, with $v_g = 0.30$, and the repaired rule keeps $\max(0.74, 0.30) = 0.74$. The fitting subsets are few enough to list: 600 + 400 (0.74), 600 + 500 (0.72), 500 + 400 + 300 (0.77), 600 + 300 (0.57), and smaller ones. The optimum is 0.77, from the three shorter passages. The repaired rule's 0.74 is within the guarantee, $0.74 \ge 0.77/2$, and below the optimum: the rule is not exact, which is why the knapsack problem needs more than a sort.

Problem 5 (the bill). At the batch prices used here, $2 and $10 per million input and output tokens for Opus 5.5 (half the list price of $4 and $20; Anthropic, 2026c, 2026d), the five twenty-passage conditions of the Opus run cost $7.52 for 750 requests, and the closed-book condition $0.17 for 150. What does a twenty-passage context cost per question, and how much of it is input?

Solution. $7.52/750 = \$0.0100$ per request, against $0.17/150 = \$0.0011$ for the question alone. A median prompt of 4,576.5 tokens costs $4{,}576.5 \times 2/10^6 = \$0.0092$ in input, about nine tenths of the mean request; a median response of 68 tokens, including thinking, costs $0.0007$. Twenty passages multiply the cost of answering by about nine, for a gain in accuracy of 8 to 16 questions in 150 depending on position. Whether that is worth it is a property of the application, but the arithmetic is the first thing context assembly decides.

What we have learned

Assembling a context is a sequence of decisions with measurable consequences. The number of passages trades a retrieval term that can only rise against reader terms that can fall, and the exact decomposition shows which is moving. Order matters for some models and not others: in our recorded run Haiku 5.5 favored the first passage and Opus 5.5 showed no effect we could detect, while both lost accuracy to distractors wherever the answer sat, the finding of Liu et al. in a weaker form. Under a model of position, the best order pairs the most probable passages with the best slots, and with equal-length passages and additive values the retriever's top choices fill the budget optimally; with unequal lengths a simple guarantee holds and exactness does not.

Everything here assumed that retrieval happens once, before the model reads. The model can instead write its own queries, retrieve again after reading, and decide when to stop. That is the subject of Retrieval in a Loop.

References