A page of this site is long: the median page of the frozen set used in this unit has 3,189 words. The dense encoder of Dense Passage Retrieval reads at most 256 word pieces, which most 200-word stretches of this site's text already exceed, so a page encoded whole is a page encoded by its opening. Ranked that way, the 200 known-item queries find their target page with a mean reciprocal rank of 0.580; ranked by the best of the site's passages, with the same encoder, 0.795. Cutting the pages helped by a wide margin. But cut how: at headings or every hundred words, into small pieces or large, with overlap or without, and with the page's title attached or not? What should the unit of retrieval be when what is retrieved will be handed to a language model?
A passage, or chunk, is a contiguous piece of a document that is indexed and retrieved as a unit; chunking is the rule that cuts documents into passages. In the systems of this unit the passage is both the unit that is scored and the unit that is read: the retriever ranks passages, and a model receives their text. The choice of passage therefore decides what can be found and what the model sees.
Here is the route. First we trace passage retrieval before language models, through Callan's three kinds of passage and Hearst's TextTiling, and the proposals made since retrieval began to feed language models. Then we prove three counting facts about windows: how many there are, which spans overlap keeps whole, and how chunk size limits the pages that fit in a fixed budget. Finally we re-cut this site's pages in many ways and measure lexical and dense retrieval over each on the site's own test queries. The article assumes Dense Passage Retrieval for the encoder and the site's test sets, and Okapi BM25 for the lexical ranker. The numbers are computed by tools/experiments/rag/passages-and-chunking.py.
Passages before language models
Passage retrieval is older than retrieval-augmented generation by decades, and its first motive was the length of documents. Callan (1994) put the problem plainly: an algorithm that "cannot distinguish a few matches scattered across a document from a dense region of matches" has difficulty with long documents and with news summaries that cover many subjects (Section 1). His experiments added passages to the INQUERY system and asked two questions that every chunking decision still faces: how should passages be defined, and how should their evidence be used?
He sorted the definitions into three classes, which this article keeps (Section 1):
- Discourse passages follow the units a writer made: sentences, paragraphs, sections.
- Semantic passages follow the subject of the text, found by an algorithm such as TextTiling.
- Window passages are a fixed number of words, regardless of content.
One might expect discourse passages to work best, since writers organize their material by content. Callan's caution was that they depend on the writer's consistency: "If writers are sloppy or rushed, or if discourse boundaries are introduced for visual presentation", the boundaries may mislead. In his collections, Wall Street Journal paragraphs were sometimes one sentence long, and Federal Register paragraphs ran from one line to thousands of words (Section 2.1). Merging short paragraphs and splitting long ones, to between 50 and 200 words, gave "bounded paragraphs". Ranking documents by their single best bounded paragraph lost 2.9% and 27.9% of average precision on 65 megabytes of Federal Register documents from TIPSTER volume 1 (Table 1) and on TIPSTER volumes 1 and 2 (Table 2, scored on each system's top 1,000 documents only), while combining document and paragraph evidence gained 1.0% and 4.3% (Section 2.2, Tables 1 and 2).
Windows did better. Callan started the first window at the first word that matched the query and began a new window every half window length, so that consecutive windows overlapped by half. Ranking each document by its best window was 20.7% better than whole-document ranking on a Federal Register collection with 300-word windows, 3.8% better on legal documents from West with 200-word windows, about equal on TIPSTER volume 1, and a combination of document and window evidence was better than either on all four collections tested (Section 2.3, Tables 3 to 6). Across sizes from 25 to 10,000 words (Table 7 shows 25 to 1,000), windows of 150 to 300 words did best, and "a window of 200 or 250 words provides consistently good results" (Section 2.3.1, Table 7). Two details of that work recur below: a document scored by its best passage alone (in preliminary experiments, recording only the best passage's contribution gave the best results; Section 2), which is the MaxP rule of Dense Passage Retrieval; and overlap, chosen so that "a small block of relevant text" is less likely to be split between two passages.
Semantic passages: TextTiling
Hearst (1997) gave the standard algorithm for semantic passages. TextTiling divides an expository text into "multi-paragraph units that represent passages, or subtopics", using only the distribution of words (Abstract). The block version works as follows (Section 5):
- Cut the text into token-sequences of $w$ words, a fixed size rather than real sentences, so that the units being compared are of equal length. Stop words count toward a sequence's length but not toward similarity.
- At every gap between two sequences, compare the block of $k$ sequences before the gap with the block of $k$ sequences after it by the cosine of their term counts. A gap inside a subtopic has similar blocks on both sides; a gap between subtopics has dissimilar ones.
- Smooth the sequence of similarities by averaging over a small window ($s$ values, $n$ times).
- Give each gap a depth score: from the gap, climb left to the nearest peak and right to the nearest peak, and add the two climbs. Hearst's procedure "assigns a depth score, the depth of the valley (if one occurs), to each token-sequence gap" (Section 5.3); read literally, the climb gives every gap a score, 0 at a peak. In her evaluation, paragraphs "with a nonpositive depth score are not eligible for boundary assignment" (Section 6.4). Deep valleys are strong changes of vocabulary.
- Draw a boundary at gaps whose depth exceeds a cutoff set from the mean $\bar s$ and standard deviation $\sigma$ of the depth scores: $\bar s - \sigma$ for a liberal cutoff and $\bar s - \sigma/2$ for a conservative one, at least three sequences apart, moved to the nearest paragraph break where one exists (Sections 5.3 and 5.5). The paper does not say whether $\sigma$ is the population or the sample standard deviation.
The defaults were $w = 20$, $k = 10$, $n = 1$ and $s = 2$ (Section 6.2). Hearst reports that the segmentation "corresponds well to human judgments" of the subtopic boundaries of 12 texts (Abstract). The algorithm uses relative depths, not absolute similarities, so that a gap with middling similarity flanked by tall peaks still counts as a boundary (Section 5.3).
Passages for language models
Retrieval for language models changed what a passage is for. In Callan's setting a passage was evidence for ranking a document; the document was still what the user received. In retrieval-augmented generation the passage itself is handed to the model, so its text has to be both findable and readable on its own, and it is paid for by the token.
The systems that founded the field used windows. Dense Passage Retrieval split each Wikipedia article into "disjoint text blocks of 100 words", 21,015,324 passages in all, and prepended to each the title of its article (Karpukhin et al., 2020, Section 4.1); the RAG models reused those passages (Lewis et al., 2020, Section 3). The title is the simplest form of context added to a passage: a window from the middle of an article does not say what the article is about, and the title does.
Two later proposals go further in opposite directions. Chen et al. (2024) index propositions, "atomic expressions within text, each encapsulating a distinct factoid and presented in a concise, self-contained natural language format", produced by a model that rewrites passages (their Propositionizer, distilled from GPT-4; Section 3), and report that this finer unit "significantly outperforms passage-level units in retrieval tasks" for dense retrievers, and that prompts built from such units improve downstream question answering "given a specific computation budget" (Abstract). An engineering post from Anthropic (2024), which this article treats as a vendor's report rather than a peer-reviewed study, prepends to every chunk a short "chunk-specific explanatory context" written by a language model that is given the whole document, "usually 50-100 tokens", before both embedding and BM25 indexing. On the authors' own test sets, with the top 20 chunks retrieved, they report that the share of relevant chunks missed fell by 35% with contextual embeddings alone, by 49% with contextual embeddings and contextual BM25, and by 67% with re-ranking added; they also note that "adding generic document summaries to chunks" had given "very limited gains" in their experiments. The site's own passages carry the cheap version of the idea: every passage is encoded as "page title. section heading. text".
What windows cost and what overlap buys
Three exact facts about windows settle the arithmetic before any measurement. Write a page as a stream of $n$ words, cut into windows of $W$ words that start every $S$ words, with $1 \le S \le W$, in the way this article's experiment cuts them: windows start at $0, S, 2S, \dots$, and the last window is the first one that reaches the end of the page, so it may be shorter than $W$. $S = W$ gives disjoint windows; $S = W/2$ gives Callan's half overlap.
Proposition 1 (how many windows). A page of $n > W$ words gives $J + 1$ windows, where $J = \lceil (n - W)/S \rceil$, and they hold $JW + (n - JS)$ words in all; a page of $n \le W$ words gives one window. As $n$ grows, the number of words indexed per word of text tends to $W/S$.
Proof. The window starting at $jS$ reaches the end when $jS + W \ge n$, that is when $j \ge (n - W)/S$. The first such $j$ is $J = \lceil (n-W)/S \rceil$, so the windows are $j = 0, \dots, J$. Each of the first $J$ has $W$ words, since $jS + W < n$ for $j < J$, and the last runs from word $JS$ to the end, $n - JS$ words. The ratio of indexed words to $n$ is $(JW + n - JS)/n = 1 + J(W - S)/n$. Since $(n - W)/S \le J < (n - W)/S + 1$, $J/n \to 1/S$ as $n \to \infty$, and the ratio tends to $1 + (W - S)/S = W/S$. $\square$
So half-overlapping windows double the index and the number of vectors to compute and store.
Proposition 2 (what overlap guarantees). Every span of $L$ consecutive words of the page lies entirely inside at least one window if $L \le W - S + 1$, and on a page of at least $W + 1$ words this is the largest such length: some span of $W - S + 2$ words lies in no window. With disjoint windows ($S = W$), a span of $L$ words, $1 \le L \le W$, that starts at a position chosen uniformly among the $W$ positions of a window is split between two windows with probability $(L - 1)/W$ (ignoring the end of the page).
Proof. Take a span of words $a, \dots, a + L - 1$ and let $j = \lfloor a/S \rfloor$. If $j \le J$, window $j$ starts at $jS \le a$ and ends after word $jS + W - 1$ (or at the page's end if it is the last window, and the span is inside the page). Since $a \le jS + S - 1$, the span's last word satisfies $a + L - 1 \le jS + S + L - 2$, and $jS + S + L - 2 \le jS + W - 1$ exactly when $L \le W - S + 1$; so the span lies in window $j$. If $j > J$, then $a \ge (J+1)S > JS$ and the last window runs from $JS$ to the end, so it contains the span. For disjoint windows, a span starting at offset $r$ within its window ($0 \le r \le W - 1$) stays inside when $r + L - 1 \le W - 1$, that is for the $W - L + 1$ offsets $r \le W - L$, and is split for the other $L - 1$; this count needs $L \le W$. For the largest length, take a page of $n \ge W + 1$ words, so that there are at least two windows ($J \ge 1$), and the span of words $S - 1, \dots, W$, which has $W - S + 2$ words and lies inside the page. Window 0 covers words $0$ to $W - 1$ and misses the span's last word; every later window starts at word $S$ or later and misses its first word. So the span lies in no window. $\square$
With $W = 100$ and $S = 50$, every span of up to 51 words is whole in some window; with disjoint 100-word windows, a 30-word statement starting at a random place is cut in two with probability $29/100$. The guarantee is about text, not about scores: a whole span inside a window does not make that window the best-scoring one.
Proposition 3 (budget and pages). If every chunk handed to a reader has at least $W$ words and the reader's budget is $B$ words, the chunks in the budget come from at most $\lceil B/W \rceil$ pages.
Proof. Suppose $c$ chunks enter the budget, filled in order, the last perhaps cut. The first $c - 1$ entered whole and left room for more, so $(c - 1)W < B$, that is $c < B/W + 1$, and $c \le \lceil B/W \rceil$. Each chunk comes from one page. $\square$
Larger chunks therefore bring fewer distinct pages into a fixed context. A ranking of whole pages fills a 1,000-word budget with one or two pages of this site, since its pages average several thousand words.
The experiment: this site's pages, re-cut
The site's own passages are the warehouse's: tools/db/chunker.py cuts each page at its second- and third-level headings and then into runs of whole Markdown blocks of 150 to 400 words, never splitting a code block, a display equation, a table or a list item (Document Delineation discusses what counts as a unit of text in general; How This Search Works describes the sections the search engine returns). In Callan's terms they are bounded discourse passages. The question here is what the other kinds of passage would have done, with the same retrievers, on the same queries.
Pages. The 679 pages of the frozen snapshot that the unit on dense retrieval fixed (commit 643484d), so that the numbers do not move as the site grows. All chunk sets are cut from the same text: the plain text of each page as the warehouse passages hold it (wiki links replaced by their text, markup, code and mathematics dropped), put back in page order, with each section's heading before its text.
Chunk sets.
- page: each page whole.
- section: each h2 or h3 section whole, the text before the first heading as one more section.
- warehouse: the site's passages.
- TextTiling: Hearst's block algorithm with her defaults ($w = 20$, $k = 10$, one pass of smoothing with $s = 2$), and boundaries at valleys deeper than $\bar s - \sigma/2$ (her conservative cutoff), at least three sequences apart. This implementation departs from Hearst in two ways. It computes depth scores only at the valleys (local minima) of the similarity curve and takes $\bar s$ and the population standard deviation $\sigma$ over those valleys alone, where Hearst scores every gap; and since the plain text has no paragraph breaks, boundaries fall at sequence gaps instead of being moved to paragraph breaks. Problem 4 shows that the reading of the depth rule can change where boundaries fall.
- windows of $W$ words every $S$ words, written W/S below, over the page's whole word stream, ignoring headings as boundaries.
Every chunk is indexed with a context prefix, the page title (and, for sections and warehouse passages, the section heading), as the site's dense vectors already are; sets marked bare drop the prefix.
Retrievers. BM25 with $k_1 = 1.2$, $b = 0.75$, the idf $\ln\big(1 + (N - n_t + 0.5)/(n_t + 0.5)\big)$ and the site's own analyzer (its tokens, stop words and Porter stems), each chunk a document; and the dense retriever of Dense Passage Retrieval, the pinned encoder sentence-transformers/all-MiniLM-L6-v2 at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41, which reads at most 256 word pieces of a chunk. A page's score is its best chunk's score, the MaxP rule. This BM25 is plain BM25 over chunks, not the site's BM25F with its fields and title bonus.
Queries and judgments. Two test sets of the site's evaluation framework (tools/search/eval/), read from git as they stood at commit 3dcfc7e so that later grading cannot move these numbers:
- the 60 topical queries, each graded page by page on a scale of 0 to 3 against a stated need. The grades were pooled from the top 20 pages of three rankers (the live BM25F, the dense ranker of the unit on dense retrieval, and their reciprocal rank fusion) and graded by a model, Claude Sonnet 5.5, in five batches (1,199, 681, 150, 95 and 9 grades;
tools/search/eval/judgment-sets.tsvdescribes each, and Hybrid Retrieval their checks). The site's owner also graded 150 calibration pairs; the first batch's agreement with those grades was an exact agreement of 0.68 and a quadratic-weighted kappa of 0.85. The framework averages every grade a pair has, so for those 150 pairs the owner's grade and two model grades are averaged; every other pair has one model grade. Judged pages outside the snapshot are dropped from each query's ideal ordering, since no chunk set can retrieve them. - the 200 known-item queries, each with one right page; all 200 targets are in the snapshot.
No query is quoted on this page.
Measures. The page rankings are scored with the framework's own measures: nDCG@10 with gain $2^g - 1$ on the topical queries, unjudged pages counted as grade 0, with J@10, the share of the top ten that is judged, beside it; condensed nDCG@10, the same measure after unjudged pages are removed from the ranking (which removes the penalty for retrieving unjudged pages, but also ignores them when they would have been relevant); and mean reciprocal rank (MRR) of the target on the known-item queries, which need no pool. A ranker that differs from the three pooled ones meets unjudged pages, and its nDCG@10 is then biased low; J@10 says by how much it can be.
The judgments are of pages, and the unit of retrieval here is a chunk. The page rankings above sidestep that, since MaxP turns chunk scores into a page ranking. The context budget measure does not: it fills a budget of 1,000 words with the best chunks in score order, the last one cut to fit, and reports the share of the budget's words that come from pages graded 2 or more, and the number of distinct pages in it. Here a chunk inherits its page's grade, with averaged grades of 2 or more counting as relevant, and the share of the budget's words from judged pages is reported beside it. This is an approximation: a chunk of a relevant page may itself be off the subject, and a chunk of an unjudged page counts as not relevant.
Results: lexical retrieval over chunks
Every chunk set holds the same text, with one difference in counting: the section and warehouse sets move each heading into the prefix (their bare versions drop the headings altogether), so their chunks' own words total 2,271,270, against 2,301,250 for the page, TextTiling and window sets, which keep headings in the text. The windows cover the 2,301,250 words of the 679 pages once each without overlap and about twice with half overlap (4,551,050 words for 100-word windows every 50, as Proposition 1 predicts for these pages; the script checks the count page by page).
| Chunk set | Chunks | Topical nDCG@10 | J@10 | Condensed nDCG@10 | Known-item MRR |
|---|---|---|---|---|---|
| page | 679 | 0.692 | 1.000 | 0.692 | 0.932 |
| section | 8,954 | 0.664 | 0.923 | 0.692 | 0.828 |
| warehouse (150 to 400 words) | 11,264 | 0.677 | 0.935 | 0.697 | 0.805 |
| TextTiling | 8,556 | 0.688 | 0.948 | 0.703 | 0.844 |
| windows 25/25 | 92,385 | 0.640 | 0.838 | 0.685 | 0.654 |
| windows 50/50 | 46,354 | 0.641 | 0.868 | 0.687 | 0.692 |
| windows 100/100 | 23,338 | 0.683 | 0.895 | 0.713 | 0.739 |
| windows 200/200 | 11,835 | 0.692 | 0.937 | 0.706 | 0.822 |
| windows 400/400 | 6,080 | 0.706 | 0.958 | 0.717 | 0.888 |
All sets have the context prefix. The median TextTiling segment has 240 words; the median section 201 and the median warehouse passage 198.
Size. On the known-item queries, whose target is one page, the mean reciprocal rank rose steadily with the size of the chunk, from 0.654 for 25-word windows to 0.888 for 400-word windows and 0.932 for whole pages. On the topical queries the trend points the same way but is weaker, and part of it is an artifact: small windows retrieve pages that no pooled ranker brought up, so J@10 falls to 0.838 for 25-word windows and their nDCG@10 is biased low. Condensed nDCG@10, which drops unjudged pages, spans only 0.685 to 0.717 across all the sets. Against the warehouse passages, in paired comparisons over the 60 topical queries, among the sets with the title prefix, only the 25-word and 50-word windows were behind with intervals that exclude zero ($-0.037$ [$-0.070$, $-0.003$] and $-0.035$ [$-0.062$, $-0.007$]), and only the 400-word windows ($+0.029$ [$+0.005$, $+0.054$]) and the 200-word windows with half overlap ($+0.023$ [$+0.002$, $+0.046$]) ahead; for condensed nDCG@10 every interval of these sets, all with the prefix, included zero. On the known-item queries the differences were large: the warehouse passages were behind whole pages by 0.127 [0.090, 0.165] in reciprocal rank.
Why would whole pages win the known-item test with plain BM25? The test's queries were written from each page's title, summary and headings (see Dense Passage Retrieval), words that are spread across a page, and a whole page is the one unit that holds all of them. A small window holds a few of the query's words and competes with every other window that holds the same few. That explains the direction of the effect for this test set; it would not carry over to queries about one fact stated in one place.
The kind of boundary. Between the discourse sets (sections and warehouse passages), the semantic set (TextTiling) and windows of similar size (200 words), the topical differences were small: against the warehouse passages, sections $-0.012$ [$-0.027$, $+0.002$], TextTiling $+0.011$ [$-0.010$, $+0.033$], 200-word windows $+0.015$ [$-0.011$, $+0.044$]. On the known-item queries TextTiling was ahead of the warehouse passages by 0.038 [0.002, 0.075] and sections by 0.023 [0.003, 0.044], both larger than the warehouse passages on average. So on this test, for BM25, the writer's headings gave no advantage over boundaries drawn from the vocabulary; for the dense encoder the comparison went the other way on the known-item queries (below).
Overlap. Half overlap doubled the index and changed little:
| Windows | Overlap | Topical nDCG@10 | Known-item MRR | Pages in the 1,000-word budget |
|---|---|---|---|---|
| 50 words | none | 0.641 | 0.692 | 8.68 |
| 50 words | half (every 25) | 0.650 | 0.684 | 7.23 |
| 100 words | none | 0.683 | 0.739 | 5.77 |
| 100 words | half (every 50) | 0.683 | 0.742 | 4.73 |
| 200 words | none | 0.692 | 0.822 | 3.40 |
| 200 words | half (every 100) | 0.699 | 0.815 | 2.90 |
With overlap, neighboring windows of the same page score alike and fill the budget together, so fewer pages enter it; the guarantee of Proposition 2, that a short span lies whole in some window, did not show up as better ranking.
The context prefix. Removing the page title (and the heading) from every chunk hurt every set, and hurt small chunks most:
| Chunk set | Topical nDCG@10 with prefix | without | Relevant share of the budget with prefix | without |
|---|---|---|---|---|
| windows 50/50 | 0.641 | 0.588 | 0.670 | 0.510 |
| windows 100/100 | 0.683 | 0.650 | 0.685 | 0.594 |
| windows 200/200 | 0.692 | 0.663 | 0.723 | 0.643 |
| warehouse | 0.677 | 0.654 | 0.717 | 0.640 |
For the warehouse passages the loss without the prefix was $-0.022$ [$-0.041$, $-0.006$] in nDCG@10 and $-0.018$ [$-0.035$, $-0.003$] in known-item reciprocal rank. A window of 50 words from the middle of a page often does not name the page's subject; the title does, and BM25 then matches it. This is the effect that DPR's prepended titles and Anthropic's generated contexts are meant to produce, measured here in its cheapest form.
Results: dense retrieval over chunks
The dense encoder reads at most 256 word pieces, and the chunk sets differ in how much of each chunk that is:
| Chunk set | Chunks | Share longer than 256 word pieces |
|---|---|---|
| page | 679 | 1.000 |
| section | 8,954 | 0.618 |
| warehouse | 11,264 | 0.597 |
| warehouse, no title or heading | 11,264 | 0.560 |
| TextTiling | 8,556 | 0.783 |
| windows 50/50 | 46,354 | 0.000 |
| windows 100/100 | 23,338 | 0.013 |
| windows 100/100, no title | 23,338 | 0.011 |
| windows 100/50 | 45,675 | 0.013 |
| windows 200/200 | 11,835 | 0.829 |
(Counted with the encoder's own tokenizer, prefix and special tokens included; 8 of the 46,354 50-word windows exceed the limit.) A chunk over the limit is represented by its beginning only.
| Chunk set | Topical nDCG@10 | J@10 | Condensed nDCG@10 | Known-item MRR |
|---|---|---|---|---|
| page | 0.605 | 0.817 | 0.642 | 0.580 |
| section | 0.684 | 1.000 | 0.684 | 0.792 |
| warehouse | 0.686 | 1.000 | 0.686 | 0.795 |
| TextTiling | 0.679 | 0.932 | 0.693 | 0.749 |
| windows 50/50 | 0.670 | 0.893 | 0.694 | 0.743 |
| windows 100/100 | 0.685 | 0.905 | 0.711 | 0.776 |
| windows 200/200 | 0.690 | 0.925 | 0.706 | 0.785 |
| windows 100/50 | 0.688 | 0.922 | 0.708 | 0.781 |
| windows 100/100, no title | 0.654 | 0.882 | 0.691 | 0.754 |
| warehouse, no title or heading | 0.663 | 0.952 | 0.675 | 0.797 |
The warehouse row is the dense ranker that was pooled for the site's judgments, so every one of its top ten pages is judged (J@10 1.000); the other sets meet unjudged pages and their plain nDCG@10 is biased low by up to the missing share. For that reason the comparisons below that favor the warehouse passages in plain nDCG@10 but not in condensed nDCG@10 should not be read as a difference between the chunkings.
Whole pages fail. Encoded whole, pages lost to the warehouse passages by 0.081 [0.042, 0.122] in nDCG@10, by 0.043 [0.010, 0.077] even in condensed nDCG@10, and by 0.215 [0.157, 0.271] in known-item reciprocal rank. That is the opening's result: an encoder that reads the first 256 word pieces of a 3,000-word page sees its introduction and nothing else. For BM25, which reads everything, whole pages were the best unit on the known-item queries; for this encoder they were the worst. The unit of retrieval cannot be chosen apart from the retriever.
Size. Among windows, larger was better for known items again (MRR 0.743, 0.776, 0.785 for 50, 100 and 200 words), but the warehouse passages, sections and 200-word windows were within a few hundredths of each other on every measure, and in condensed nDCG@10 the 100-word windows were ahead of the warehouse passages by 0.026 [0.004, 0.047]. The encoder reads 100-word windows whole, while it reads only the beginning of most warehouse passages; on these judged pages that did not translate into a clear difference in the ranking of pages.
Boundaries. TextTiling's segments, the longest of the sets and 78.3% truncated, were behind the warehouse passages on the known-item queries by 0.046 [0.008, 0.086] and not distinguishable from them on the topical ones. With an encoder of limited reach, a boundary that produces long segments is a cost, whatever its quality.
The context prefix. Without the title and heading, the warehouse passages changed by $-0.023$ [$-0.049$, $+0.002$] in nDCG@10, an interval that just includes zero, and nothing in known-item reciprocal rank, but the share of the 1,000-word budget that came from pages graded 2 or more fell from 0.757 to 0.655, a difference of $-0.101$ [$-0.152$, $-0.057$], and the budget drew on more pages (4.57 against 3.65). Without the title, more off-subject passages from other pages reached the top of the ranking, though the page rankings by best passage moved less. For 100-word windows the title mattered more: without it, nDCG@10 fell by 0.031 [0.007, 0.055] and the relevant share of the budget by 0.101 [0.044, 0.160].
Overlap. Half-overlapping 100-word windows ranked pages as the disjoint ones did (nDCG@10 $+0.003$ [$-0.010$, $+0.017$], known-item reciprocal rank $+0.005$ [$-0.017$, $+0.027$]) at twice the vectors, 45,675 against 23,338. They raised the relevant share of the budget by 0.028 [0.007, 0.051] while drawing on fewer pages (3.65 against 4.73), because neighboring windows of a good page tend to score high together; the guarantee of Proposition 2 did not show up as better ranking for this retriever either.
What fills the context
The two page measures above rank pages. A reader receives chunks. The budget measure fills 1,000 words with the best chunks in score order:
| Retriever | Chunk set | Pages in the budget | Share from pages graded 2 or more | Share from judged pages |
|---|---|---|---|---|
| BM25 | page | 1.10 | 0.732 | 1.000 |
| BM25 | warehouse | 4.03 | 0.717 | 0.988 |
| BM25 | windows 50/50 | 8.68 | 0.670 | 0.929 |
| BM25 | windows 400/400 | 2.47 | 0.749 | 0.997 |
| dense | page | 1.12 | 0.715 | 0.967 |
| dense | warehouse | 3.65 | 0.757 | 1.000 |
| dense | windows 50/50 | 5.77 | 0.741 | 0.976 |
| dense | windows 200/200 | 3.20 | 0.736 | 0.983 |
Small chunks spread the budget over many pages, as Proposition 3 allows and large chunks forbid: BM25's 50-word windows brought in 8.68 pages on average, its whole pages 1.10. A context of many pages is not automatically a better one; whether it is depends on what the reader does with breadth, which is the question of Assembling the Context and, for repeated material, of Result Diversification. What the shares say is narrower: under these judgments, between about two thirds and three quarters of the words of every budget came from pages graded 2 or more, and the differences between chunkings were of a few hundredths except where the prefix was removed.
What is proved here, what is cited, and what the experiment shows
Proved on the page. How many windows a page of $n$ words gives for a window $W$ and stride $S$, how many words they hold, and that the index grows by the factor $W/S$ (Proposition 1, also checked page by page on this site); that every span of at most $W - S + 1$ words lies whole in some window, that on pages of more than $W$ words this is the largest guaranteed span, and that disjoint windows split an $L$-word span ($L \le W$) at a random offset with probability $(L - 1)/W$ (Proposition 2, with Problem 2 as an instance); that chunks of at least $W$ words fill a budget of $B$ words from at most $\lceil B/W \rceil$ pages (Proposition 3).
Cited. Callan's three kinds of passage, his paragraph and window experiments and their results (Callan, 1994); the TextTiling algorithm, its defaults and its evaluation (Hearst, 1997); DPR's and RAG's 100-word passages and titles (Karpukhin et al., 2020; Lewis et al., 2020); propositions as retrieval units and their results (Chen et al., 2024); contextual retrieval and its reported results, as a vendor's report (Anthropic, 2024).
Shown by the experiment. On the 679 frozen pages of this site, with plain BM25 and one small dense encoder, on 60 topical queries judged by a model and 200 known-item queries written by a model: the size of the chunk and the presence of the page title mattered; overlap changed nothing detectable in the page rankings; the kind of boundary (headings, vocabulary, fixed windows of similar size) made no detectable difference on the topical queries and moved known-item MRR by up to about 0.04, upward for BM25 and downward for the dense encoder with TextTiling; whole pages were the best unit for BM25 on the known-item queries and the worst for the dense encoder, which reads only their beginnings. Topical comparisons between chunk sets that retrieve unjudged pages carry the bias that J@10 measures. None of this decides how the site's search should cut its pages: the site's ranker is BM25F over whole pages with fields, which was not varied here.
Worked problems
These problems are posed on small exact cases; the unit has no course assignments. The script recomputes every number and checks Propositions 1 and 2 on them.
Problem 1 (counting windows). A page has 1,000 words. How many windows of 100 words does it give with a stride of 50, and with a stride of 100, and how many words does each index hold?
By Proposition 1 with $n = 1{,}000$ and $W = 100$: for $S = 50$, $J = \lceil 900/50 \rceil = 18$, so there are 19 windows holding $18 \times 100 + 1{,}000 - 18 \times 50 = 1{,}900$ words, 1.9 times the page; the limit $W/S = 2$ is approached as pages grow. Here the first 50 words (0 to 49) and the last 50 (950 to 999) each lie in one window only, and every other word in two: $2 \times 1{,}000 - 100 = 1{,}900$. For $S = 100$, $J = 9$, 10 windows and exactly 1,000 words.
Problem 2 (the overlap that keeps a span whole). With windows of 100 words, what is the largest stride that keeps every span of 60 consecutive words whole in some window? With disjoint windows, how likely is a 30-word span to be split?
By Proposition 2 a span of $L$ words is always whole when $L \le W - S + 1$, so for $L = 60$ we need $S \le 100 - 60 + 1 = 41$. The bound is tight. With $S = 42$, take the span of words $42j + 41$ to $42j + 100$, inside the page. Window $j$ covers words $42j$ to $42j + 99$ and misses the span's last word; window $j + 1$ starts at word $42j + 42$ and misses its first; earlier windows end earlier and later ones start later. The span lies in no window. With disjoint windows a 30-word span is split for 29 of the 100 possible starting offsets, a probability of $29/100$.
Problem 3 (pages in a budget). A reader takes 1,000 words of chunks. At most how many pages can they come from with chunks of at least 400 words, and how many did the 400-word windows of the experiment bring in?
By Proposition 3, at most $\lceil 1{,}000/400 \rceil = 3$ pages: two whole chunks of 400 words leave 200, which the third fills in part. The experiment's 400-word windows brought 2.47 pages on average, below the bound, because consecutive chunks often come from the same page. (A page's last window can be shorter than 400 words, so the bound applies to these windows only approximately.)
Problem 4 (a TextTiling boundary by hand). After smoothing, the similarities at ten successive gaps are 0.62, 0.55, 0.31, 0.40, 0.58, 0.60, 0.52, 0.45, 0.50 and 0.57. Where do the conservative and the liberal cutoffs put boundaries, under this article's reading of the depth rule (depths at valleys only, population standard deviation) and under the readings of Hearst's text?
This article's reading. The valleys are at gaps 3 (0.31) and 8 (0.45), counting from 1. From gap 3 the curve climbs left to 0.62 and right to 0.60, so its depth is $(0.62 - 0.31) + (0.60 - 0.31) = 0.60$. From gap 8 it climbs left to 0.60 and right to 0.57, a depth of $(0.60 - 0.45) + (0.57 - 0.45) = 0.27$. The two depths have mean $\bar s = 0.435$ and population standard deviation $\sigma = 0.165$. The conservative cutoff is $\bar s - \sigma/2 = 0.3525$: gap 3 exceeds it and becomes a boundary, gap 8 does not. The liberal cutoff is $\bar s - \sigma = 0.27$, which gap 8's depth equals without exceeding, so the liberal rule also draws only the boundary at gap 3.
Hearst's procedure read literally. Climbing from every gap gives depths 0, 0.07, 0.60, 0.20, 0.02, 0, 0.08, 0.27, 0.07 and 0: a gap on a slope climbs only on the side that rises. Their mean is 0.131 and their population standard deviation 0.178, so the cutoffs are 0.042 and $-0.047$. Taking gaps deepest first, leaving out gaps of depth 0, and keeping those at least three gaps from a boundary already drawn, both cutoffs give boundaries at gaps 3 and 8. A third reading of "(if one occurs)" gives the valleys their depths and every other gap 0, with the statistics over all ten gaps: mean 0.087, population standard deviation 0.189, cutoffs $-0.008$ and $-0.102$, and again boundaries at gaps 3 and 8. Finally, with the valley depths but the sample standard deviation (0.233), the conservative cutoff is 0.318 and still draws only gap 3, while the liberal cutoff is 0.202, which gap 8 exceeds, so the liberal rule draws two boundaries. Every reading that takes the statistics over all ten gaps gives two boundaries; the answer depends on the reading, which is why the experiment states its own.
What we have learned
Passages were first a way to score long documents by their best part: Callan found that windows of a few hundred words, overlapping by half, ranked documents as well as or better than the whole documents did, and the two kinds of evidence together better still, and Hearst gave a way to cut text where its vocabulary changes. For a language model the passage is also what is read, so it must be findable and self-explanatory and it is paid for by the word. The counting is exact: windows of $W$ words every $S$ words cost $W/S$ times the text, overlap of $W - S$ words keeps every span of up to $W - S + 1$ words whole, and chunks of $W$ words fill a budget of $B$ words from at most $\lceil B/W \rceil$ pages. On this site's pages, the size of the passage and the page title attached to it changed retrieval more than overlap did, and the choice between headings, vocabulary shifts and fixed windows mattered only on the known-item queries, by about 0.04 and in opposite directions for the two retrievers; the right size depended on the retriever, since BM25 read whole pages well and a 256-word-piece encoder could not.
What is retrieved must then be arranged for the reader: how many passages, in what order, and how many from one page. That is the subject of Assembling the Context. The other end of the range of units, retrieval of a single next token, is the subject of Nearest Neighbor Language Models.
References
- Anthropic (2024). Introducing Contextual Retrieval. Engineering post, 19 September 2024 (a vendor's report). https://www.anthropic.com/research/contextual-retrieval record
- Callan, J. P. (1994). Passage-level evidence in document retrieval. In Proceedings of SIGIR 1994, 302–310. https://doi.org/10.1007/978-1-4471-2099-5_31. Author's copy: https://www.cs.cmu.edu/~callan/Papers/callan794.pdf record · 2 pages
- Chen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Zhang, H., and Yu, D. (2024). Dense X retrieval: What retrieval granularity should we use? In Proceedings of EMNLP 2024, 15159–15177. https://doi.org/10.18653/v1/2024.emnlp-main.845 record
- Hearst, M. A. (1997). TextTiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1), 33–64. https://aclanthology.org/J97-1003/ record
- Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP 2020, 6769–6781. https://doi.org/10.18653/v1/2020.emnlp-main.550 record · 15 pages
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2005.11401. record · 6 pages