The assistant that reads this site, the owner's chatbot, has one search tool: the site's JSON API, which takes a query and returns pages with their best passages. Asked what constant reciprocal rank fusion adds to each rank, and whether fusing the site's BM25F ranker with a dense retriever helped on the site's own test queries, it cannot send the question to one search and be done: the constant is on one page and the measurement on another, and the second half of the question needs a query of its own, which the answer to the first half suggests. The retrieval-augmented systems of Open-Domain Question Answering and the RAG models retrieve once, with the input as the query. What changes when the language model writes the queries itself, and decides when and how often to retrieve? This article answers with the methods published up to November 2026; it is a dated survey, and its account of current practice should be read as of that month.
Retrieval in a loop here means any arrangement in which a language model's own output decides some part of retrieval: what the query is, whether to retrieve at all, how many times, and when to stop. The model is no longer only the reader of a ranked list. It is also the author of the queries that produce the list.
Here is the route. First we see three ways in which one retrieval falls short, and set out a single loop of four decisions that every method fills in differently. Then come the methods in order of how much they hand to the model: HyDE, which only rewrites the query, with a proof that its averaged query vector is a fusion of ordinary rankings; IRCoT and FLARE, which retrieve again during generation; Self-RAG, which trains the model to decide; and search as a tool call. Finally the site's own API serves as the worked case, traced by hand, given to two models on three questions, and used to try HyDE on the site's known-item queries. The article assumes Dense Passage Retrieval for the dense retriever and How This Search Works for the site's search engine and API. The numbers are computed by tools/experiments/rag/retrieval-in-a-loop.py from the recorded runs.
Why one retrieval is not always enough
The design of open-domain question answering retrieves once. The question is the query, the retriever returns its top $k$ passages, and the reader answers from them. Three kinds of failure come from that single step, and each method in this article answers one of them.
The first is a mismatch of wording. A question and the passage that answers it are different kinds of text: the question names what is missing, and the passage states it. A lexical ranker needs shared words, and a dense retriever needs an encoder trained to map the two kinds of text close together. Without training data for that mapping, the question is a poor query for its own answer.
The second is a dependence between steps. Trivedi et al. (2023, Figure 1) give the question "In what country was Lost Gravity manufactured?" The passage about the roller coaster Lost Gravity says that it was made by Mack Rides, and only a passage about Mack Rides says that the company is from Germany. That second passage shares no informative word with the question. It can be found only after the first has been read, with a query that the first passage supplies.
The third is retrieval that should not happen at all. Some inputs need no outside text, yet a retriever always returns $k$ passages, relevant or not. Asai et al. (2024, Abstract) write that "indiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary, or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation."
In all three the remedy is the same in kind: the model takes part in retrieval. It writes the query, it decides when to retrieve again, or it decides whether to retrieve at all.
The loop in general
Every method below fits one loop. At each step the system holds the input $q$, the passages collected so far, and the text generated so far, $y_{<t}$ in the unit's notation. It makes four decisions:
- Whether to retrieve at this step.
- What query to send: the input itself, a sentence the model has just written, a passage the model has imagined, or a query the model writes on purpose.
- What to keep: all retrieved passages, the best few, or those the model judges relevant.
- When to stop: after a fixed number of steps, when the model writes an answer, or when it stops asking.
Single-step retrieval is the special case that retrieves once, at the start, with $q$ as the query, keeps the top $k$, and stops. The methods differ in which decisions they hand to the model, and in whether the model is trained for them or only prompted.
| Method | When it retrieves | What the query is | Training |
|---|---|---|---|
| HyDE (Gao et al., 2023) | once | a generated passage | none |
| IRCoT (Trivedi et al., 2023) | after each reasoning sentence | the last reasoning sentence | none (few-shot prompts) |
| FLARE (Jiang et al., 2023) | when a drafted token's probability is below a threshold | the drafted sentence | none in its prompted form |
| Self-RAG (Asai et al., 2024) | when the model emits a Retrieve token | the input and the previous segment | fine-tuned generator and critic |
| Search as a tool | whenever the model calls it | whatever the model writes | varies |
Writing a better query: hypothetical documents
The idea in plain words. If a question is a poor query for its answer, let a language model write an answer first, true or not, and search with that. A made-up answer is a passage of the same kind as the passages in the collection, so a document encoder compares like with like.
Gao, Ma, Lin and Callan (2023) call the method HyDE, for Hypothetical Document Embeddings. Their setting is zero-shot dense retrieval: no relevance judgments of any kind are available for training. An encoder trained without labels, Contriever, maps documents to vectors. HyDE never compares the query with the documents through a trained query encoder. It prompts an instruction-following model with a fixed instruction, for web search "Please write a passage to answer the question" followed by the question (Gao et al., 2023, Appendix A.1), samples $N$ passages $\hat d_1, \dots, \hat d_N$, and encodes each with the document encoder $E_P$. The query vector is the mean of those vectors and of the query's own (their Equation 8):
$$\vec{v} = \frac{1}{N + 1}\Big(E_P(q) + \sum_{j=1}^{N} E_P(\hat d_j)\Big).$$
Passages are ranked by the dot product $\vec{v} \cdot \vec{p}$. The generated passages may state false facts; the authors expect the encoder's "dense bottleneck" to act as a "lossy compressor", keeping the pattern of a relevant document and dropping invented details (Sections 1 and 3.2). Their generator was InstructGPT through the text-davinci-003 endpoint, sampled at temperature 0.7 (Section 4.1).
What averaging the vectors does. The mean vector looks like a new kind of query. It is in fact a fusion of $N + 1$ ordinary dense rankings.
Proposition 1 (HyDE's mean vector is score fusion). For every passage $p$, $$\vec{v} \cdot \vec{p} = \frac{1}{N+1}\Big(s(q, p) + \sum_{j=1}^N s(\hat d_j, p)\Big),$$ where $s(x, p) = E_P(x) \cdot \vec{p}$ is the dense score of $p$ for the text $x$. So ranking by $\vec v$ is ranking by the sum of the $N + 1$ dense scores, which is CombSUM over those rankings without normalization (Rank Fusion). Scaling $\vec v$ by any positive constant, normalizing it to unit length for example, does not change the order.
Proof. The dot product is linear in its first argument: $\big(\sum_i a_i \vec{x}_i\big) \cdot \vec{p} = \sum_i a_i (\vec{x}_i \cdot \vec{p})$. Apply this with $a_i = 1/(N+1)$ to the $N + 1$ vectors $E_P(q), E_P(\hat d_1), \dots, E_P(\hat d_N)$. For a constant $c > 0$, $(c\vec v) \cdot \vec p = c\,(\vec v \cdot \vec p)$, and multiplying every score by the same positive number keeps their order. $\square$
The proposition explains why HyDE needs no new index and no new search code, and it places HyDE among known methods: query expansion by generated text, fused at the level of scores. It also says what the averaging can and cannot do. A passage's fused score lies between its smallest and its largest score among the $N + 1$ rankings, and a passage that another passage beats in every one of the summed rankings cannot overtake it. It does not say that a passage ranked low by every part stays low: in ranks, it can come first. With $N = 1$, if the query's dense scores for three passages are 0.9, 0.1 and 0.6 and the generated passage's are 0.1, 0.9 and 0.6, the third passage is second in both rankings and first in the fused one (0.6 against 0.5 and 0.5). In general, with $m = N + 1 \ge 2$ rankings, give a passage the score 0.6 in all of them, and for each ranking $i$ give $r$ other passages the score 1 in ranking $i$ and 0 in the others: the passage is ranked $r + 1$ in every ranking and first after fusion, with 0.6 against $1/m$. The statement about overtaking is also about passages. When pages are ranked by their best passage, as on this site, it fails for pages: page A with one passage scoring 0.7 in both rankings, and page B with two passages scoring 0.9 and 0 in the first ranking and 0 and 0.9 in the second, has B ahead of A in both rankings and A ahead after fusion, 0.7 against 0.45.
What was reported. On the TREC Deep Learning 2019 queries, nDCG@10 was 44.5 for Contriever alone, 50.6 for BM25 and 61.3 for HyDE built on Contriever; on the 2020 queries the figures were 42.1, 48.0 and 57.9 (Gao et al., 2023, Table 1). HyDE was also ahead of its base encoder in nDCG@10 on all seven low-resource BEIR collections that the paper reports, and ahead of the multilingual mContriever in MRR@100 on the four Mr. TyDi languages (Sections 4.3 and 4.4, Tables 2 and 3). The authors state two limitations: HyDE "relies on real-time generation from LLMs and therefore may not be suitable for tasks that demand high throughput or low latency", and HyDE "may prefer certain content in its generation and therefore bias the final search results" ("Limitations").
Retrieval interleaved with reasoning
The idea in plain words. For a question that needs several facts in sequence, let the model reason one sentence at a time, and after each sentence search with that sentence. Each new sentence names what the next search needs.
IRCoT (Trivedi, Balasubramanian, Khot and Sabharwal, 2023) is that loop, run with a prompted model and no training. It first retrieves $K$ paragraphs with the question as the query, then alternates two steps (Section 3 and Figure 2):
- Reason. The model receives the question, every paragraph collected so far and the reasoning sentences written so far, and writes the next sentence of a chain of thought. Only the first sentence it generates is kept.
- Retrieve. The last sentence is the query; $K$ more paragraphs are added to the collection.
The loop stops when a sentence contains "answer is" or when the number of steps reaches a limit (8 in the experiments). The collected paragraphs, capped (at 15) so that the prompt still fits a few demonstrations, are the retrieval result, and a separate reader answers from them. The retriever is BM25 in Elasticsearch (Section 4). On four multi-step question sets (HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC), the authors report that IRCoT with GPT-3 (code-davinci-002) improved retrieval by up to 21 points and question answering by up to 15 points over retrieval with the question alone, with similar gains for the much smaller Flan-T5 models, and fewer factual errors in the generated reasoning (Abstract and Section 1). In the Lost Gravity example the first reasoning sentence, "The Lost Gravity was manufactured by Mack Rides", is a query that contains the name the second retrieval needs.
Jiang et al. (2023) move the trigger from the end of every sentence to the model's confidence. Their method, FLARE (forward-looking active retrieval augmented generation), drafts the next sentence without retrieval. If every token of the draft has a probability of at least a threshold $\theta$, the draft is kept. Otherwise the draft becomes the query, either with its uncertain tokens masked or turned into explicit questions, and the sentence is generated again from the retrieved documents (Section 3.2). With $\theta = 0$ retrieval never happens, and with $\theta = 1$ it happens before every sentence. The design rests on the authors' premise that language models "tend to be well-calibrated", so that low probability often signals missing knowledge; one of the papers they cite for it is the subject of Language Models (Mostly) Know What They Know.
Retrieval the model decides to do
The idea in plain words. Teach the model to emit, in its own output, a token that says whether to retrieve, and further tokens that grade what it retrieved and what it wrote. Decoding can then use those grades.
Self-RAG (Asai, Wu, Wang, Sil and Hajishirzi, 2024) adds four kinds of reflection tokens to a model's vocabulary (Table 1 of the paper):
| Token | Reads | Values | Meaning |
|---|---|---|---|
| Retrieve | input, previous output | yes, no, continue | whether to retrieve now |
| IsRel | input, passage | relevant, irrelevant | whether the passage helps |
| IsSup | input, passage, output | fully supported, partially supported, no support | whether the output is supported by the passage |
| IsUse | input, output | 5, 4, 3, 2, 1 | how useful the output is |
Generation proceeds one segment at a time; in the experiments a segment is a sentence. At each segment the model predicts Retrieve. If it says yes, the retriever returns passages, the model writes one candidate segment per passage in parallel, grades each with IsRel, IsSup and IsUse, and keeps the best by a score that adds to the segment's generation probability a weighted sum of the grades' normalized probabilities, with a beam search over segments using the same score (Section 3.3). Instead of the hard yes or no, retrieval can be triggered whenever the probability of Retrieve = yes passes a threshold.
The tokens are learned, not prompted. A critic model was trained on reflection tokens assigned by GPT-4, and the critic then labelled the generator's training data offline, so that the generator learns to produce the tokens itself (Section 3.2). The paper appeared at ICLR 2024, and the settings that follow are those of its published version (the arXiv version gives the same). The generators were Llama 2 at 7 and 13 billion parameters, trained on 150,000 instruction and output pairs; the retriever was Contriever fine-tuned on MS MARCO; at inference the weights of IsRel, IsSup and IsUse were 1.0, 1.0 and 0.5, the retrieval threshold 0.2 for most tasks, and the segment beam width 2, with the top five documents by default (Section 4.3). The authors report that Self-RAG outperformed ChatGPT and retrieval-augmented Llama 2 chat models on open-domain question answering, reasoning and fact verification (Abstract); on long-form answers with citations (ASQA) its citation precision and recall were higher than those of every baseline except ChatGPT, which it exceeded in citation precision only (Section 5.1); citation precision and recall are the subject of Attribution and Citation.
Search as a tool call
The idea in plain words. Give the model a search engine the way a program is given a library function: a name, a description and a list of arguments. The model decides when to call it and with what query, reads the result, and continues.
Three papers marked the way.
- WebGPT (Nakano et al., 2021) fine-tuned GPT-3 to answer long questions by operating a text-based web browser whose searches went to the Bing Web Search API, collecting references as it browsed. It was trained by imitating human demonstrations of the task, then improved with rejection sampling against a reward model of human preferences. On questions from the ELI5 set its best model's answers were preferred to those of the human demonstrators 56% of the time and to the highest-voted Reddit answers 69% of the time (Abstract).
- Toolformer (Schick et al., 2023) let a model teach itself where calls help. A GPT-J model of 6.7 billion parameters was prompted to propose calls, written inline as text such as
[QA("Where was Joe Biden born?")], at positions in ordinary text. A call was kept only if giving the model the call and its result lowered the model's weighted loss on the following tokens by at least a threshold $\tau_f$, compared with the better of no call and the call without its result: $L_i^- - L_i^+ \ge \tau_f$ (Section 2). The model was then fine-tuned on the text with the kept calls. One of its tools was BM25 search over a Wikipedia dump (Section 3). - ReAct (Yao et al., 2023) prompted a model to alternate written reasoning with actions in a small Wikipedia interface of three operations:
search[entity](the first five sentences of the page, or five similar titles),lookup[string](the next sentence containing the string) andfinish[answer](Section 3.1). The authors note that this interface is "significantly weaker than state-of-the-art lexical or neural retrievers"; they chose it to force retrieval through explicit reasoning.
As of November 2026 the pattern is built into model APIs. A tool is declared with a name, a description and a JSON Schema for its input. The model's reply may contain a structured request to call a tool with arguments; the calling program runs the tool and returns its output in the next message; and the model continues until it answers. The runs recorded below use exactly this protocol, with content blocks of type tool_use in the model's replies and tool_result in the program's. A search tool in this setting is a retriever whose query, depth and stopping rule all belong to the model, and whose results the model reads in full. That is the sense in which Building effective agents lists retrieval, tools and memory as the "augmentations" of the basic model.
The worked case: this site's API
This site publishes such a tool. How This Search Works describes its two endpoints; here they are called as an assistant would call them, on the site as it stood at commit 817b13a (759 pages), run locally through Flask's test client, so that nothing was sent over the network and the responses can be reproduced. The script records every response in data/retrieval-in-a-loop-trace.json.
The tool definitions. The site's /openapi.json declares four operations; the two for retrieval are searchSite (GET /api/search, parameters q, section and limit) and readPage (GET /api/page/{slug}, parameters offset and max_chars). The description of searchSite is the only place where the site tells a model how to search, and it says: "Use a few specific keywords rather than a whole question, and search again with different words if the results miss." In the terms of the loop above, the tool leaves all four decisions to the model and advises it on the second.
One question, four calls. Take the running question. A loop that follows the advice can make these calls:
| Call | What came back | Characters of JSON |
|---|---|---|
searchSite, q reciprocal rank fusion constant, limit 5 |
475 matching pages; first Rank Fusion (score 7.863), then Hybrid Retrieval (7.481); two passages per page | 16,507 |
readPage, slug rank-fusion, max_chars 4,000 |
sections 0 and 1 of 15; next_offset 2 |
3,876 |
readPage, slug rank-fusion, max_chars 4,000, offset 2 |
sections 2 to 5; next_offset 6 |
5,177 |
searchSite, q hybrid retrieval dense BM25F fusion, limit 5 |
282 matching pages; first Hybrid Retrieval (13.201), with its passages "This site" and "The judged comparison" | 17,119 |
The first search puts the page that defines the constant first, with its section "Reciprocal rank fusion" as the first passage. The second half of the question needs a second query, because the first result list was chosen for the words of the first half. This is the IRCoT pattern done by hand: the second query names what the first answer pointed to.
The advice, tested. Sent as a whole sentence (what is the constant in reciprocal rank fusion and why is it sixty), the same search matched all 759 pages, because free text is a disjunction of its words and every page contains some of them; its first result was Hybrid Retrieval, with Rank Fusion second. Misspelled (recriprocal rank fusoin), the query was corrected to reciprocal rank fusion and the response said so in corrected_from, so a model can see what was actually searched. A request without q returned status 400 with {"error": "missing q"}, and an unknown slug status 404 with {"error": "no page no-such-page"}. Errors come back as data that a model can read and act on.
Two models with the two tools
The trace shows what the tools return. It does not show what a model does with them. Two models were therefore given the two operations and three questions about the site. This is a small recorded demonstration, not an experiment: three questions, one run per model, no repetition. Its value is that every call, response and answer is recorded and can be inspected.
Setup. The tool definitions were built from the site's own /openapi.json (name, summary and description, and each parameter's schema and description), and calls were executed against the site at commit 817b13a through the test client. The system prompt was: "You answer questions about the website omnipedia.tech using its two tools. Search with a few specific keywords, read pages when the passages are not enough, and cite the URLs of the pages you used. Answer in a short paragraph." The questions, written for this page:
- On this site, what constant does reciprocal rank fusion add to each rank, and did fusing the site's BM25F ranker with a dense retriever help on the site's own evaluation queries? Cite the pages.
- Which of this site's pages explains the efSearch parameter of HNSW graphs, and what recall@10 does it report at efSearch 50? Cite the page.
- What weight does this site's search engine give the title field, and which page derives the BM25F function it uses? Cite the pages.
The models were claude-opus-5-5 (effort "low") and claude-haiku-5-5 (default settings), called on 18 November 2026 through the unit's recording harness, at most 8 turns and 4,000 output tokens per turn. The runs are data/runs/loop-tools-opus.jsonl.gz and loop-tools-haiku.jsonl.gz.
| Model | Question | Turns | Searches | Page reads | Input tokens, all turns | Cost (USD) |
|---|---|---|---|---|---|---|
| Opus 5.5 | 1 | 4 | 1 | 2 | 48,791 | 0.211 |
| Opus 5.5 | 2 | 2 | 1 | 0 | 8,375 | 0.040 |
| Opus 5.5 | 3 | 2 | 1 | 0 | 7,887 | 0.042 |
| Haiku 5.5 | 1 | 2 | 2 | 0 | 14,231 | 0.002 |
| Haiku 5.5 | 2 | 2 | 1 | 0 | 8,397 | 0.001 |
| Haiku 5.5 | 3 | 3 | 3 | 0 | 34,556 | 0.004 |
What the models did. Both followed the tool's advice and searched with keywords, not with the questions. Haiku split the first question into two searches issued in the same turn (reciprocal rank fusion, then BM25F dense retriever fusion evaluation); Opus searched once with the words of both halves and then read the Hybrid Retrieval page. Both answered every question with the facts the pages state: the constant 60 with the page Rank Fusion; the fusion's nDCG@10 of 0.771 against BM25F's 0.716 on the topical queries, from Hybrid Retrieval; 0.852 at efSearch 50 from the HNSW page; a title weight of 12, and the wiki page BM25F for the derivation.
Where they went wrong. The errors are of the kind that Attribution and Citation measures: right numbers, wrongly described.
- The HNSW page prints "0.852 (0.851 to 0.853)" and states that its tables give "the median over the 30 builds with the first and third quartiles in parentheses". Opus called the bracket "a range of 0.851 to 0.853 across seeds" and Haiku a "95% interval". Neither is what the page says, although the passage that both models received from the search, the section "efSearch", contains the sentence "The interquartile ranges are narrow" a few lines after the table.
- Haiku wrote that the fusion "also did not beat BM25F on the site's known-item queries, where BM25F's lead was clear", citing Dense Passage Retrieval. The first half is what that page reports: on the known-item queries "fusion is not ahead: its MRR is 0.916 against BM25F's 0.942" (and Hybrid Retrieval, which Haiku also cited, calls the corresponding difference one that the 200 queries "do not distinguish from zero"). The second half, "BM25F's lead was clear", belongs on that page to BM25F against the dense ranker alone. The answer merged two comparisons in one clause.
- Opus asked
readPagefor the slughybrid-retrieval#this-sitewithmax_chars8,000, a slug the site does not have. The recording script put the slug into the request path without percent-encoding it, so the request was/api/page/hybrid-retrieval#this-site?max_chars=8000: the#began a fragment that swallowed both#this-siteand the query string, and the site returned the page Hybrid Retrieval from its first section with the default limit, 29,143 characters. A client that encodes path parameters, as OpenAPI requires, would have senthybrid-retrieval%23this-siteand received status 404 with an error naming the missing page, which the model could have read and corrected. The model then read on from offset 13 with a second call. The fault is shared: the model invented an anchor in a slug, and the harness turned that into a silent success. The recording is kept as it was made; the script's encoding was not changed afterwards, since that would change a recorded run. - For the third question Opus found the title weight on the page RankNet, which restates it, and said in its answer that it had not opened the table in How This Search Works that the passage pointed to. Haiku, after a third search, cited How This Search Works directly.
What a call costs. The tokens of the table are input tokens summed over turns, because every turn of a conversation carries everything before it. For Opus's first question the input grew from 879 tokens on the first turn to 7,287, 18,757 and 21,868: the first search result, 17,729 characters of JSON, made the next request 6,408 tokens larger, 68 of them the model's own call. A loop's cost is therefore roughly quadratic in its number of turns, the arithmetic of Problem 2 below in another form, and the budget question of Assembling the Context arises at every step of a loop, not once.
HyDE on this site's known-item queries
The site's evaluation framework has 200 known-item queries, each written for one target page and scored by the rank of that page; Dense Passage Retrieval describes the set and its bias toward lexical rankers (the queries were written from the pages' titles, summaries and headings). Because each has one right answer, it needs no relevance pool, and a new ranker can be scored on it without new judgments. That makes it the set on which HyDE can be tried here. The topical queries are not used: their judgments were pooled from three other rankers, and the pages a HyDE ranker would bring up are partly unjudged.
Setup. The frozen snapshot of the unit on dense retrieval: 679 pages at commit 643484d, cut into 11,264 passages and encoded by sentence-transformers/all-MiniLM-L6-v2 at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41; a page is ranked by its best passage. All 200 targets are in the snapshot; the query file is read as it stood at commit 3dcfc7e. For each query, HyDE's web-search instruction was sent verbatim with the query in place of the question: four times to claude-haiku-5-5 and once to claude-opus-5-5, both with effort "low" and at most 1,024 output tokens, through the Message Batches API on 18 November 2026 (runs loop-hyde-haiku-1024 and loop-hyde-opus-1024; generation cost 0.18 and 2.00 US dollars). The encoder reads at most 256 word pieces of each generated passage, so only the beginning of a long passage counts. Many replies reached the 1,024-token limit: 316 of Haiku's 800 and 168 of Opus's 200, the median being 911.5 and 1,024 output tokens, thinking included (the median thinking was 210 and 145.5 tokens). For the dense rows this matters little, since the encoder reads only the first 256 word pieces of a passage; for the BM25F row below, which appends the whole first Haiku passage, the cut passages are part of what was appended. A first run with a 400-token limit and, for Haiku, no effort setting (runs loop-hyde-haiku and loop-hyde-opus) returned no text for 378 of Haiku's 800 requests (370 because thinking used the whole budget, 8 refusals) and for 33 of Opus's 200 (31 and 2); it is kept in data/runs/ and was replaced by the runs reported here. Generations that came back with no text (8 refusals and 11 replies cut off before any text among Haiku's 800; 2 refusals and 1 cut-off reply among Opus's 200) were left out of the means. Two other refusals returned some text and were used as written: a 1,100-character passage from Haiku, and the two characters ** from Opus, which were encoded and averaged like any passage (in the row of one Opus passage they are the whole representation of that query). 4 of Haiku's and 3 of Opus's queries had no usable passage and kept the query's own vector. The query texts are not reproduced here.
| Query representation | MRR | Target first | Target in top 10 | Reciprocal rank minus the query alone [95% interval] |
|---|---|---|---|---|
| query alone (dense) | 0.795 | 0.705 | 0.950 | |
| one Haiku passage | 0.835 | 0.755 | 0.975 | +0.040 [−0.005, +0.085] |
| four Haiku passages | 0.833 | 0.735 | 0.970 | +0.038 [−0.001, +0.078] |
| four Haiku passages and the query (Equation 8) | 0.854 | 0.765 | 0.965 | +0.058 [+0.025, +0.093] |
| one Opus passage | 0.816 | 0.715 | 0.965 | +0.021 [−0.033, +0.073] |
| one Opus passage and the query | 0.847 | 0.755 | 0.975 | +0.052 [+0.017, +0.087] |
| BM25F, the site's search, query alone | 0.942 | 0.905 | 1.000 |
Intervals are 95% percentile intervals of the mean difference from a paired bootstrap over the 200 queries (10,000 resamples, seed 20261113, the framework's convention).
What the run shows. On these queries, averaging generated passages with the query raised the dense retriever's mean reciprocal rank from 0.795 to 0.854 with Haiku's four passages and to 0.847 with Opus's one, and both intervals exclude zero. Generated passages without the query were not distinguishable from the query alone: the intervals of all three such rows include zero. That fits Proposition 1: the query is one ranking in the sum, and here it was a good one. Here one Opus passage was not more useful than one Haiku passage (0.816 against 0.835, a difference not tested), and one Opus passage with the query scored close to four Haiku passages with it, at about eleven times the cost.
What it does not show. The site's own BM25F, on the same queries, places the target first for 0.905 of them and reaches an MRR of 0.942, far above every dense variant; known-item queries written from a page's own words favor a lexical ranker, as Hybrid Retrieval found for the dense ranker without HyDE. Adding the first Haiku passage to the BM25F query as extra words did not help that ranker: its MRR fell to 0.695, a change of −0.247 [−0.297, −0.199], because a passage of some 300 words turns a precise query into a long disjunction. None of this says whether HyDE would serve the site's readers; it says that on a test set of 200 model-written queries, with one small encoder, generated passages were a useful addition to the query for dense retrieval and a harmful one for lexical retrieval. Each HyDE query also costs a generation before the search can start, the latency limitation its authors state.
What is proved here, what is cited, and what the runs show
Proved on the page. That HyDE's averaged query vector ranks passages exactly as the sum of the dense scores of the query and the generated passages, CombSUM without normalization, and that rescaling it changes nothing (Proposition 1); the counts of Problems 2 to 5.
Cited. HyDE's method, instruction and results (Gao et al., 2023); IRCoT's algorithm and results (Trivedi et al., 2023); FLARE's trigger and its premise (Jiang et al., 2023); Self-RAG's tokens, training and settings (Asai et al., 2024); WebGPT's setup and preference rates (Nakano et al., 2021); Toolformer's filter and tools (Schick et al., 2023); ReAct's interface (Yao et al., 2023).
Shown by the recorded runs. On this site's API at one commit: what each call returns and costs in characters and, in the model runs, in tokens; that two models, given three questions, used keyword queries, found the facts asked for, mostly on the right pages, and still misdescribed what a number on a page meant; and, on the 200 known-item queries with one encoder, that HyDE's averaged vectors improved the dense ranking of the target page while generated passages alone did not measurably, and that generated text appended to a lexical query hurt it. These are observations of single recorded runs with models that cannot be run at a fixed temperature, not estimates of how these methods behave in general.
Worked problems
These problems are posed on small exact cases and on the recorded trace; the unit has no course assignments. The script recomputes every number.
Problem 1 (HyDE's mean vector as fusion). In two dimensions, a query encodes to $(1, 0)$ and two generated passages to $(0.6, 0.8)$ and $(0, 1)$. Two collection passages have unit vectors $\vec p_1 = (0.8, 0.6)$ and $\vec p_2 = (0.28, 0.96)$. Rank them by HyDE's Equation 8 and check Proposition 1; then rank them by the two generated passages alone.
The dense scores are: for the query, $0.8$ and $0.28$; for the first generated passage, $0.48 + 0.48 = 0.96$ and $0.168 + 0.768 = 0.936$; for the second, $0.6$ and $0.96$. The mean vector is $\vec v = \tfrac13(1 + 0.6 + 0,\ 0 + 0.8 + 1) = (0.5333, 0.6)$, so $\vec v \cdot \vec p_1 = 0.4267 + 0.36 = 0.7867$ and $\vec v \cdot \vec p_2 = 0.1493 + 0.576 = 0.7253$. The averages of the three scores are $(0.8 + 0.96 + 0.6)/3 = 0.7867$ and $(0.28 + 0.936 + 0.96)/3 = 0.7253$, the same numbers, as Proposition 1 says, and $p_1$ is first. Without the query the averages are $(0.96 + 0.6)/2 = 0.78$ and $(0.936 + 0.96)/2 = 0.948$, and $p_2$ is first. Whether the query is counted can change the order, because it is one more ranking in the sum.
Problem 2 (what a loop reads). An IRCoT-style loop retrieves $K = 4$ paragraphs for the question and 4 more after each of $S = 3$ reasoning steps, with no duplicates. How many paragraphs does it collect, and how many paragraph readings do the reasoning steps make in all, if each step reads every paragraph collected so far?
It collects $K(S + 1) = 16$ paragraphs. Reasoning step $j$, for $j = 1, 2, 3$, comes after $j$ retrievals and reads $4j$ paragraphs: 4, 8 and 12, so 24 readings in all. In general the steps read $K(1 + 2 + \dots + S) = KS(S + 1)/2$ paragraphs. The collection grows linearly with the number of steps and the reading quadratically, which is why IRCoT caps the collection.
Problem 3 (FLARE's trigger). A drafted sentence has token probabilities 0.92, 0.71, 0.38, 0.88 and 0.95. Is retrieval triggered with $\theta = 0.4$? With $\theta = 0.35$?
Retrieval is triggered when any token's probability is below $\theta$. The smallest is 0.38, which is below 0.4 and above 0.35. With $\theta = 0.4$ the sentence is regenerated from retrieved documents; with $\theta = 0.35$ the draft is kept.
Problem 4 (Toolformer's filter). For a candidate call at position $i$, the weighted losses of the following tokens are 2.10 with no call, 2.05 with the call but no result, and 1.70 with the call and its result. For which thresholds $\tau_f$ is the call kept?
$L_i^- = \min(2.10, 2.05) = 2.05$ and $L_i^+ = 1.70$, so $L_i^- - L_i^+ = 0.35$, and the call is kept exactly when $\tau_f \le 0.35$. The minimum matters: compared with no call at all the gain would be $2.10 - 1.70 = 0.40$, but $0.05$ of it comes from merely writing the call, and the result is not credited with that.
Problem 5 (reading a page in pieces). readPage adds whole sections while the characters of their text and heading stay within max_chars, and always returns at least one. In the trace, the first sections of Rank Fusion have texts of 1,688, 1,172 and 1,225 characters, with headings "Rank Fusion" (11 characters), "Scores that cannot be added" (27) and "Score-based fusion" (18). Why did the first call with max_chars 4,000 stop after two sections, and why did the second stop at offset 6?
The running sums are $1{,}688 + 11 = 1{,}699$, then $1{,}699 + 1{,}172 + 27 = 2{,}898$, then $2{,}898 + 1{,}225 + 18 = 4{,}141$, which exceeds 4,000. So the third section is left for the next call, and next_offset is 2. The second call, from offset 2, took sections of 1,243, 1,667 (text 1,654 and heading "Normalization"), 359 (342 and "Rank-based fusion") and 689 (674 and "The Borda count") characters, 3,958 in all; the next section would have passed 4,000, so next_offset was 6. A model that pages through a document pays per call in turns and per character in context, and the endpoint lets it choose the trade.
What we have learned
A retrieval-augmented model need not take one list of passages. It can write its own query (HyDE), search again after each step of its reasoning (IRCoT), when its confidence drops (FLARE), or when trained tokens say so (Self-RAG), and with a search tool it decides everything: the queries, how far to read and when to stop. HyDE's averaged query vector is score fusion of ordinary dense rankings, so it needs no new machinery, and a passage that another passage beats in every one of the summed rankings cannot overtake it. On this site, two models given the site's own search API found the facts asked for with keyword queries and still described some of them wrongly, and every turn of a loop carried the whole context again. HyDE's generated passages, averaged with the query, helped the site's dense retriever on the known-item queries, and the site's lexical search remained far ahead of it on those queries.
A model that writes queries and reads results still has to decide what to keep in its context and in what order, and whether what it then writes is supported by what it read. Those are the subjects of Assembling the Context and Attribution and Citation; how such systems are scored is the subject of Evaluating Retrieval-Augmented Systems.
References
- Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. (2024). Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In Proceedings of ICLR 2024. arXiv:2310.11511. record
- Gao, L., Ma, X., Lin, J., and Callan, J. (2023). Precise zero-shot dense retrieval without relevance labels. In Proceedings of ACL 2023 (Volume 1: Long Papers), 1762–1777. https://doi.org/10.18653/v1/2023.acl-long.99 record
- Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G. (2023). Active retrieval augmented generation. In Proceedings of EMNLP 2023, 7969–7992. https://doi.org/10.18653/v1/2023.emnlp-main.495 record
- Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021). WebGPT: Browser-assisted question-answering with human feedback. arXiv:2112.09332. record
- Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. (2023). Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). arXiv:2302.04761. record
- Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. (2023). Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of ACL 2023 (Volume 1: Long Papers), 10014–10037. https://doi.org/10.18653/v1/2023.acl-long.557 record
- Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In Proceedings of ICLR 2023. arXiv:2210.03629. record