Hybrid Retrieval

How a lexical ranker and a dense retriever are combined into one first stage, and when the combination pays. The two rankers' lists are measured for overlap on Cranfield, where they share fewer than half their top ten and far fewer of their wrong answers than lexical rankers do. Score interpolation is analysed exactly, with proofs that normalization changes only the weight's scale, that a document beaten on both scores cannot rise, and that the fused ranking changes at finitely many weights. Interpolation and reciprocal rank fusion are compared with cross-validation and paired tests, and the three rankers are compared on this site's topical queries after pooling all three, where the fusion is ahead of the site's lexical ranker and the dense ranker alone is not, with the limits of that test set stated. Five worked problems cover breakpoints, normalization, dominance, fusion by hand and the recall ceiling.

Cranfield query 185 asks for experimental studies on panel flutter. BM25 puts abstract 856, some experimental studies of panel flutter at mach 1.3, first and has six relevant abstracts in its top ten. The dense retriever of Dense Passage Retrieval puts a review of panel flutter first and has three. Fused by reciprocal rank fusion, the two lists give a third list with four relevant abstracts in its top ten, worse than BM25's and better than the dense list's. Over all 225 queries, the same fusion beats both of its inputs by margins the paired tests do not attribute to chance: 0.051 in nDCG@10 over BM25 ($t = 5.59$) and 0.037 over the dense retriever ($t = 4.17$). Why does a combination that loses on this query win on average, how should the two lists be combined, and what does the same fusion do on this site's own search?

Hybrid retrieval is first-stage retrieval that combines a lexical ranker, which scores the query's own terms, with a learned ranker, which scores a representation the query and document were mapped to, into one result list. The usual learned partner is a dense retriever, and that is the case worked here. The combination can be made after retrieval, by fusing two ranked lists, or inside one model that holds both kinds of evidence; this article treats the first. The question it answers is practical: given a BM25-like engine and a bi-encoder, which combination to use and how to tell whether it helps.

Here is the route. First we measure how differently the two rankers err, which is the condition under which fusion can help at all. Then we set out the two families of combination: rank fusion, already treated in Rank Fusion, and score interpolation, which needs normalization and a weight. Three results about interpolation are proved: normalization only rescales the weight, a document beaten on both scores never rises above the document that beats it, and the fused ranking is piecewise constant in the weight. They make the tuning of the weight exact. Then come the Cranfield results with cross-validation and paired tests, what published experiments found, and finally this site, where the three rankers are compared on judged queries. The article assumes Rank Fusion and Dense Passage Retrieval. The numbers are computed by tools/experiments/neural-ranking/hybrid-retrieval.py.

Do the two rankers err differently?

Fusion can only help where the inputs disagree in a useful way. Rank Fusion reports the conditions of Lee (1997) and of Vogt and Cottrell (1998): the rankers combined should be good separately, retrieve many of the same relevant documents, and retrieve different nonrelevant ones. It measured them for three lexical rankers on Cranfield and found the nonrelevant overlap high, between 0.72 and 0.87, and no fusion of the three beat BM25.

The same measurements for BM25 and the dense retriever, on the top 100 of each list for all 225 queries:

Measure BM25 and dense
Mean number of abstracts shared by the two top tens 4.23 of 10
Queries whose two top tens share nothing 2
Mean Jaccard overlap of the two top 100s 0.306
Lee's overlap of relevant abstracts, top 100 0.870
Lee's overlap of nonrelevant abstracts, top 100 0.433

Lee's overlap coefficient for two runs is $2c/(n_1 + n_2)$, with $c$ the number of documents both retrieved and $n_1$, $n_2$ the numbers each retrieved, counted here over the relevant documents (or the nonrelevant ones) and summed over the queries, as Rank Fusion counts it. The relevant overlap, 0.870, is a little below the lexical pairs' 0.930 to 0.965. The nonrelevant overlap, 0.433, is about half of theirs. The two rankers agree on most of what is relevant and disagree on most of what is not, the pattern Lee called the unequal overlap property and the one under which agreement between lists is evidence of relevance.

The relevant abstracts split this way. Cranfield has 1,612 query-abstract pairs judged relevant. Both rankers place 999 of them in their top 100; 115 are only in BM25's top 100 and 183 only in the dense retriever's; 315 are in neither. So each ranker finds relevant abstracts that the other misses, which is the skimming effect, and the 999 they share are the agreement that RRF rewards, the chorus effect.

Example (agreement is not redundancy). It is tempting to read "they share fewer than half their top ten" as "one of them must be wrong about the rest". Query 185 shows otherwise. BM25's top ten holds 856, 857, 766, 390, 858, 1008, 658, 864, 859 and 391; the dense top ten holds 658, 864, 856, 1008, 878, 857, 390, 627, 948 and 391. They share seven. BM25's three that the dense list lacks, 766, 858 and 859, are all relevant; the dense list's three that BM25 lacks, 878, 627 and 948, are not. On this query the disagreement is BM25's gain. On the 225 queries together the dense retriever's extra relevant abstracts outnumber BM25's, 183 to 115.

Two ways to combine

Fusing ranks

Reciprocal rank fusion gives document $d$ the score $\sum_i 1/(k_{\mathrm{RRF}} + r_i(d))$, summed over the lists $i$ that retrieved it, with $r_i(d)$ its rank in list $i$ and $k_{\mathrm{RRF}} = 60$ (the constant written $k$ in Rank Fusion; here $k$ is a cutoff). That article proves the properties used below. RRF is unaffected by any strictly increasing rescaling of a ranker's scores, so it needs no normalization. The constant sets how far down agreement between lists outweighs one list's first place: with two lists, down to rank $k_{\mathrm{RRF}} + 1$. It also defines CombSUM and CombMNZ, which add normalized scores. The rest of this section is about the other family, which uses the scores.

Interpolating scores

A convex combination of the two scores is

$$f_\alpha(d) = \alpha\, \hat s(d) + (1 - \alpha)\, \hat l(d), \qquad 0 \le \alpha \le 1,$$

where $\hat l(d)$ is the lexical score of document $d$ for the query and $\hat s(d)$ the dense score, each after a per-query normalization. At $\alpha = 0$ the ranking is the lexical ranker's, at $\alpha = 1$ the dense retriever's. The weight $\alpha$ is the only parameter. Two normalizations are in common use, each computed over the documents of one query:

  • min-max: $\hat l(d) = (l(d) - \min l)/(\max l - \min l)$, the same for $s$;
  • z-score: $\hat l(d) = (l(d) - \mu_l)/\sigma_l$, with $\mu_l$ and $\sigma_l$ the mean and standard deviation of the query's lexical scores.

Some systems add raw scores with a weight on one of them. The dense passage retriever of Karpukhin et al. (2020) ranked the union of the two top-2,000 lists by $\mathrm{BM25}(q, p) + \lambda\, s(q, p)$ with $\lambda = 1.1$ (Dense Passage Retrieval reports the results). That is a convex combination of raw scores after dividing by $1 + \lambda$.

There is a practical difference from rank fusion that the definitions hide. Interpolation needs both scores of every document it ranks. A document retrieved by only one ranker has no score from the other, and systems fill it in some way: Ma et al. (2021), replicating DPR, give such a document the lowest score the other ranker returned; Bruch, Gai and Ingber (2023) compute the missing score exactly. On Cranfield the question does not arise: the dense retriever scores all 1,400 abstracts, and BM25's score of an abstract with no query term is exactly 0, so every interpolation below is computed over the whole collection with true scores.

What can be proved about interpolation

Three propositions describe how $f_\alpha$ behaves as the weight moves. Each concerns a single query.

Normalization only rescales the weight

Here is the idea in one sentence: any normalization that multiplies a query's scores by a positive number and adds a constant gives the same set of rankings as any other such normalization, reached at different weights.

Proposition 1. Fix a query. Let the normalized scores be $\hat s(d) = a\, s(d) + b$ and $\hat l(d) = c\, l(d) + e$ with $a, c > 0$, and let the normalized scores $\tilde s(d) = a' s(d) + b'$ and $\tilde l(d) = c' l(d) + e'$ come from a second such normalization. For every $\alpha \in [0, 1]$ there is an $\alpha' \in [0, 1]$ such that $\alpha \hat s + (1 - \alpha) \hat l$ and $\alpha' \tilde s + (1 - \alpha') \tilde l$ order the documents identically.

Proof. Expand the first combination:

$$\alpha \hat s(d) + (1 - \alpha) \hat l(d) = \alpha a\, s(d) + (1 - \alpha) c\, l(d) + \big(\alpha b + (1 - \alpha) e\big).$$

The last term is the same for every document, so dropping it changes no order. Write $A = \alpha a$ and $B = (1 - \alpha) c$, both nonnegative and not both zero. Then $A s(d) + B l(d) = (A + B)\big(\beta s(d) + (1 - \beta) l(d)\big)$ with $\beta = A/(A + B) \in [0, 1]$, and dividing by the positive constant $A + B$ changes no order. So the first combination ranks as the raw combination with weight $\beta$. The same argument applied to the second normalization shows that weight $\alpha'$ ranks as the raw combination with weight $\beta' = \alpha' a'/(\alpha' a' + (1 - \alpha') c')$. The map $\alpha' \mapsto \beta'$ is continuous and increasing on $[0, 1]$, from 0 to 1, so some $\alpha'$ gives $\beta' = \beta$, and then the two combinations rank alike. $\square$

Bruch, Gai and Ingber (2023, §4) reduce any such positive affine normalization to the raw scores with a changed weight, and state the min-max against z-score case as their Lemma 4.2; the proof above writes out the general case, including no normalization at all ($a = c = 1$). The map from $\alpha$ to $\beta$ depends on the query's score ranges, and that is what the proposition does not say: a single weight chosen for a whole query set means a different weight for each query under each normalization. So the choice of normalization does matter for a fixed weight, and it does not matter for a weight chosen per query.

Check on Cranfield. For each query, the best nDCG@10 that any weight achieves is the same under min-max, under z-score and with raw scores: the script asserts equality for all 225 queries, and the mean of these per-query bests is 0.487 under all three. The best single weight for all queries differs: it gives 0.420 under min-max, 0.421 under z-score and 0.418 with raw scores.

A document beaten on both scores cannot rise

Proposition 2. Fix a query and a cutoff $k$. If at least $k$ documents have both a higher lexical score and a higher dense score than document $d$, then $d$ is outside the top $k$ of $f_\alpha$ for every $\alpha \in [0, 1]$.

Proof. Let $j$ be one of those documents: $\hat l(j) > \hat l(d)$ and $\hat s(j) > \hat s(d)$ (a positive affine normalization keeps both inequalities). For $\alpha \in [0, 1]$,

$$f_\alpha(j) - f_\alpha(d) = \alpha\big(\hat s(j) - \hat s(d)\big) + (1 - \alpha)\big(\hat l(j) - \hat l(d)\big) > 0,$$

since both differences are positive and the weights $\alpha$ and $1 - \alpha$ are nonnegative and sum to 1. So each of the $k$ documents is above $d$ at every weight, and $d$ is at best at rank $k + 1$. $\square$

The converse fails: a document beaten on both scores by fewer than $k$ others may still never reach the top $k$, and worked problem 3 has one. The proposition is a filter. On Cranfield, with $k = 10$, it leaves between 12 and 43 of the 1,400 abstracts per query, 22.6 on average, as the only ones any weight can bring into the top ten.

The ranking changes at finitely many weights

Proposition 3. Fix a query. As $\alpha$ runs over $[0, 1]$, the order of $n$ documents under $f_\alpha$ changes at no more than $n(n-1)/2$ weights, and between consecutive such weights it is constant.

Proof. Each $f_\alpha(d)$ is a linear function of $\alpha$. Two documents $u$ and $v$ swap order only where $f_\alpha(u) = f_\alpha(v)$, that is where

$$\alpha\big(\hat s(u) - \hat l(u) - \hat s(v) + \hat l(v)\big) = \hat l(v) - \hat l(u).$$

If the coefficient of $\alpha$ is not zero, this holds at exactly one $\alpha$; if it is zero, the two lines are parallel and either never meet or coincide everywhere, and in neither case do they swap. So each of the $n(n-1)/2$ pairs contributes at most one weight, and away from those weights no pair changes order. $\square$

Together, Propositions 2 and 3 make the weight's effect on nDCG@10 computable exactly. For each query, keep the documents Proposition 2 allows, compute the crossing weights of every pair of them, and evaluate nDCG@10 once on each interval between consecutive crossings. The result is a step function of $\alpha$. On Cranfield the median query has 12 weights at which its nDCG@10 changes, and the most has 49. The mean over a set of queries is again a step function, whose steps are the union of the queries' steps, 3,113 pieces for all 225 queries. Its maximum is found exactly, with no grid. Ties in score are broken by document number throughout, as everywhere in the unit.

Mean nDCG@10 over Cranfield's 225 queries as the weight on the dense score goes from 0 (BM25 alone) to 1 (the dense retriever alone), for min-max and z-score normalization, computed exactly as step functions, with RRF's value as a horizontal line. Notice the broad plateau from about 0.5 to 0.8, where the two normalizations are within a few thousandths of each other and of RRF, and that no weight rises clearly above RRF's line.
Mean nDCG@10 over Cranfield's 225 queries as the weight on the dense score goes from 0 (BM25 alone) to 1 (the dense retriever alone), for min-max and z-score normalization, computed exactly as step functions, with RRF's value as a horizontal line. Notice the broad plateau from about 0.5 to 0.8, where the two normalizations are within a few thousandths of each other and of RRF, and that no weight rises clearly above RRF's line.

Combination on Cranfield

Query 185. Its nDCG@10 under min-max interpolation takes 26 values as $\alpha$ moves from 0 to 1. It is 0.807, BM25's value, up to $\alpha = 0.006$, falls, not quite steadily (it rises from 0.690 to 0.704 at $\alpha = 0.149$), to 0.582 on $(0.520, 0.564)$, and ends at 0.290, the dense value, above $\alpha = 0.989$. For this query the best weight is no weight at all: any weight above 0.006 costs something. RRF's list scores 0.521.

The comparison over all 225 queries uses the setup of Dense Passage Retrieval: nDCG@10 with gain $5 - \text{grade}$, recall with any graded abstract counted as relevant. A weight chosen on the same queries it is scored on is optimistic, so the interpolation rows use two-fold cross-validation on the unit's seeded split of the queries into a development half of 112 and a test half of 113: the best weight on the development half, found exactly as above, is applied to the test half, and the reverse, and the 225 resulting values are averaged. The best mean on a half is reached on a short interval of weights, and the weight applied is that interval's midpoint: under min-max the development half's interval is $(0.676, 0.678)$, so $\alpha = 0.677$, and the test half's is $(0.758, 0.759)$, so $\alpha = 0.759$; under z-score the two halves chose 0.774 and 0.584, and with raw scores 0.987 and 0.971. RRF, CombSUM and CombMNZ fuse the two top-1,000 lists, with each list's scores min-max normalized over the documents it retrieved, as Rank Fusion does.

Ranker nDCG@10 Recall@10 Recall@100 Recall@1000
BM25 0.368 0.397 0.737 0.952
Dense 0.382 0.416 0.772 0.989
RRF, $k_{\mathrm{RRF}} = 60$ 0.419 0.455 0.789 0.989
CombSUM 0.418 0.460 0.794 0.990
CombMNZ 0.418 0.460 0.789 0.989
Interpolation, min-max, weight cross-validated 0.415
Interpolation, z-score, weight cross-validated 0.416
Interpolation, raw scores, weight cross-validated 0.414
Interpolation, best weight per query (oracle) 0.487

The first three rows repeat the numbers of Dense Passage Retrieval, computed again here. The interpolation rows report nDCG@10 only, because the weights were tuned for it.

Paired against BM25 over the 225 queries, every fusion is ahead, and by amounts the tests do not attribute to chance:

Fusion minus BM25, nDCG@10 Difference $t$ Randomization $p$ 95% bootstrap interval Better / worse
RRF +0.051 5.59 < 0.0001 [0.033, 0.068] 131 / 58
CombSUM +0.050 6.29 < 0.0001 [0.034, 0.065] 133 / 53
Interpolation, min-max, cross-validated +0.047 4.99 < 0.0001 [0.029, 0.066] 131 / 62
Interpolation, z-score, cross-validated +0.048 5.14 < 0.0001 [0.029, 0.066] 126 / 64

The randomization test flips the signs of the 225 differences 20,000 times (seed 1); none of the flips reached the observed mean, so the estimated $p$ is 0 out of 20,000, and the $t$-test gives $p < 10^{-5}$ for each row. The bootstrap resamples queries 10,000 times (seed 20261113).

Several things follow from the two tables.

  • Every fusion beats both inputs. RRF is 0.051 above BM25 and 0.037 above the dense retriever ($t = 4.17$). This is the result that the three lexical rankers of Rank Fusion did not produce, and the overlap table of the first section is the reason the literature gives.
  • A tuned weight does not beat RRF here. Cross-validated min-max interpolation is 0.003 below RRF ($t = -0.69$, $p = 0.49$; better on 72 queries, worse on 85). The difference is too small for these 225 queries to show in either direction.
  • Tuning is optimistic. The best single weight for all 225 queries reaches 0.420 when it is scored on the queries that chose it, and 0.415 when it must be chosen on the other half. The development half's own best was 0.448 and the test half's 0.397: the halves differ in difficulty, and their best weights differ by 0.08, from 0.677 to 0.759.
  • Raw scores need an extreme weight. With raw scores the best weight is about 0.99. Proposition 1 explains why: BM25's scores for a query span tens of units (the top score, which is the range since the lowest is 0, has a median of 24.2 over the 225 queries, and is 18.0 for query 185) and the cosines less than 1, so a raw sum is BM25 unless nearly all the weight is on the dense side. Worked problem 2 does the conversion for query 185.
  • The oracle is far above everything. If each query could have its own best weight, the mean would be 0.487. The weight that suits a query varies. For 54 queries the best value is already reached next to $\alpha = 0$, BM25's end (query 185 is one), and for 63 next to $\alpha = 1$, the dense end; the two counts share the 17 queries for which the weight changes nothing. For the other 125, only weights strictly inside do best. No fusion here knows in advance which kind of query it has.

The RRF constant. With two lists the constant matters little in the middle of its range: nDCG@10 is 0.404 at $k_{\mathrm{RRF}} = 0$ and at 1, 0.419 at 10, 0.417 at 30, 0.419 at 60 and at 100, and 0.417 at 1,000. Small constants reward a single first place too much, as Rank Fusion proves.

The recall ceiling of a fusion

Fusion re-orders what its inputs retrieved, and that bounds what it can find.

This is the recall ceiling that Multi-Stage Ranking proves for a cascade (its Proposition 1), with the union of the lists as the candidate set; the two-line proof is repeated for completeness.

Proposition 4. Let a fusion rank only documents in the union $U$ of its inputs' lists. Then for every cutoff $k$, its recall@$k$ is at most the share of the relevant documents that lie in $U$.

Proof. The fused top $k$ is a subset of $U$, so the relevant documents in it are a subset of the relevant documents in $U$. Dividing both counts by the number of relevant documents keeps the inequality. $\square$

On Cranfield the union of the two top-100 lists holds 0.834 of the relevant abstracts on average, while RRF's own top 100 holds 0.789. The bound is not reached, because the union has up to 200 documents and the fused list keeps 100. In a system that fuses two cut lists, the depth of each input sets the ceiling, which is the recall ceiling of a cascade discussed in Multi-Stage Ranking.

What published experiments found

All results in this section are the papers' own, as the papers report them.

  • DPR and its replication. Karpukhin et al. (2020) found their BM25-plus-DPR combination better than DPR alone on some datasets and worse on others (Dense Passage Retrieval gives the table). Ma et al. (2021) replicated the work with their own BM25, which they report was stronger than the original paper's, and concluded the opposite about hybrids: "hybrid techniques yield statistically significant gains", about three points of top-20 accuracy over the best DPR results, averaged over five question-answering collections. They tuned the weight by grid search over $[0, 2]$ in steps of 0.05.
  • Dense retrievers and BM25. Wang, Zhuang and Zuccon (2021) interpolated BM25 with two BERT-based dense retrievers, RepBERT and ANCE, on MS MARCO and the TREC 2019 and 2020 Deep Learning passage tasks, varying the weight from 0 to 1 in steps of 0.1. They report that interpolation was needed for the dense retrievers to be effective, unlike for a BERT cross-encoder re-ranker, and their analysis attributes it to the dense retrievers being good at strong relevance signals and poor at weaker ones, which BM25 supplies. They also computed a per-query oracle weight, as this article does, and report that it delivers large, significant gains over the other settings.
  • Out of domain. Chen et al. (2022) fused BM25 and a deep retriever with RRF, chosen because it has no parameter to tune on a new domain. On three out-of-domain datasets they report average relative gains in recall of 20.4% over the deep retriever and 9.54% over BM25.
  • Interpolation against RRF. Bruch, Gai and Ingber (2023) compared the two with BM25 and the same all-MiniLM-L6-v2 model that this unit uses, on MS MARCO and eight BEIR datasets, measuring nDCG@1,000 (nDCG@100 on the two smallest). With the weight tuned on the validation splits of their three in-domain datasets ($\alpha = 0.8$ on the dense score, after min-max normalization with the theoretical minimum of each score in place of the observed one), interpolation scored 0.454 on MS MARCO against 0.425 for RRF with constant 60, 0.309 for BM25 and 0.441 for the dense model, and they report it significantly ahead of RRF on every dataset in their Table 2. They also found that RRF tuned with a separate constant for each ranker generalized poorly across datasets, and that interpolation is sample-efficient: a small set of training examples suffices to tune its one weight for a new domain.

The Cranfield result sits between these. Fusion helps, as all of them found. Interpolation with a tuned weight was not ahead of RRF here, as it was in Bruch and colleagues' experiments, at a different cutoff, on far larger collections and with the weight tuned on validation queries. Cranfield's 225 queries leave an interval of about a hundredth either side of the difference, which covers both outcomes.

This site

The site's search engine is a lexical ranker, BM25F over whole pages with weighted fields (How This Search Works describes it). The plan for this unit asked this article to report whether the site should add a dense stage. This section reports the measurement. The decision is the owner's, and it involves more than effectiveness, as the last part of the section sets out.

Three rankers are registered in the evaluation framework (tools/search/eval/rankers.py) and run on the site's pages as they stood at commit 5abda21, the commit at which the pool below was judged: 743 pages, cut into 13,031 passages by the same chunker and encoded by the same model as in Dense Passage Retrieval.

  • baseline, the live BM25F search;
  • dense-minilm, each page scored by its best passage's cosine with the query;
  • hybrid-rrf, RRF with $k_{\mathrm{RRF}} = 60$ of the first two, top 100 of each. Nothing in it was tuned on the site: the constant is the one Rank Fusion takes from Cormack, Clarke and Büttcher.

The test collection and its judgments

The site has 60 topical test queries, each with a stated need, split at random into 29 tune and 31 held queries, and 200 known-item queries, each written for one target page. All of them were written by a model (Claude), not by readers: the topical queries from the library's thread questions and the course topic outlines, without running the search, and the known-item queries from each target page's title, summary and headings. The evaluation's documentation lists this among its known limits and expects the queries to be easier for a lexical ranker than real queries probably are; for the known-item set it measures the reuse: on average 89% of a query's terms occur in its target's summary.

Topical judgments were made by pooling: for each query, the top 20 pages of each of the three rankers were collected (1,910 pairs), together with 65 pairs kept from earlier pools that none of the three now ranks in its top 20, 1,975 in all, and every pair was graded from 0 to 3 against the evaluation's judging guide. Dense Passage Retrieval could not compare the rankers because the pool then held only BM25F's top 20. Now each ranker's top 20 is in the pool and judged, for every query, so nDCG@10 is computed on judged pages only for all three. The grades come from several judgment sets (tools/search/eval/judgment-sets.tsv):

  • claude-sonnet-5-5-skim: a model (Claude Sonnet 5.5) grading 1,199 of the 1,200 pairs of the first pool, BM25F's top 20 (the other was left for the owner and graded in the refresh batch), from each page's front matter, headings and the sections bearing on the need. Its agreement with the owner's own grades was measured on 150 calibration pairs: exact agreement 0.68, quadratic-weighted kappa 0.85.
  • mdb (the owner) and claude-sonnet-5-5-full (the same model reading each page in full): the same 150 calibration pairs.
  • claude-sonnet-5-5-refresh: 95 pairs, those a later re-pooling of BM25F added and the one pair left by the skim batch.
  • claude-sonnet-5-5-hybrid: the 681 pairs that pooling dense-minilm and hybrid-rrf, with the baseline re-pooled on the pages of commit 5abda21, added. Its protocol differs from the skim batch's: most clear zeros were judged from the opening, headings and passages located by search, and five torn calls first left at the higher grade were lowered. Its agreement with a full-reading judge was measured on 60 of these pairs drawn by grade (24, 18, 12 and 6 of grades 0 to 3), so the figures describe that sample: exact agreement 0.80, quadratic-weighted kappa 0.90.

Where a pair has grades from several sets, metrics.qrels() averages them. The judge's agreement with the owner was measured, not imposed: nothing adjusted the model's grades to the owner's.

The judged comparison

The measures are the evaluation framework's own (tools/search/evaluate.py): nDCG@10 with gain $2^g - 1$, R@20 as the share of the pages graded 2 or more in the pool that a ranker places in its top 20, and 95% percentile bootstrap intervals over queries with 10,000 resamples (seed 20261113), the same resamples for every ranker of a split. The script recomputes every value with the framework's functions.

Ranker nDCG@10, all 60 Tune (29) Held (31) R@20, all 60
baseline (BM25F) 0.716 [0.664, 0.763] 0.707 0.725 0.831
dense-minilm 0.681 [0.631, 0.728] 0.634 0.725 0.807
hybrid-rrf 0.771 [0.728, 0.809] 0.744 0.796 0.894

Paired against the baseline, query by query (difference, 95% bootstrap interval, and the share of resamples in which the difference is not positive):

Minus BM25F nDCG@10, all 60 Tune (29) Held (31) R@20, all 60
dense-minilm −0.035 [−0.084, +0.015], 0.914 −0.073 [−0.155, +0.010] +0.001 [−0.052, +0.053] −0.025 [−0.090, +0.039]
hybrid-rrf +0.055 [+0.023, +0.087], 0.001 +0.037 [−0.018, +0.087] +0.071 [+0.037, +0.109] +0.063 [+0.021, +0.107]

J@10, the share of the top ten that has a judgment, is 1.000 for all three rankers.

What the tables show, and no more:

  • The hybrid is ahead of BM25F on the topical queries. Over all 60 its nDCG@10 is higher by 0.055, the interval excludes zero, and the per-query differences give the same verdict by Student's test ($t = 3.39$, $p = 0.001$; better on 41 queries, worse on 19). On the 31 held queries the difference is 0.071 with an interval clear of zero; on the 29 tune queries it is 0.037, and that interval includes zero. Recall at 20 is higher by 0.063. Since nothing in the hybrid was tuned on these queries, the split matters less here than for a tuned variant; it is reported because the framework reports it.
  • The dense ranker alone is not distinguishable from BM25F. Over all 60 it is 0.035 lower, with an interval that includes zero ($t = -1.37$, $p = 0.17$; better on 24, worse on 36); it is lower on the tune queries and level on the held ones. The hybrid is ahead of the dense ranker by 0.090 ($t = 6.32$; better on 48 queries, worse on 12).
  • The pattern is Cranfield's. There too the dense retriever alone was level with the lexical ranker and the fusion was ahead of both.

Known items. The 200 known-item queries each have one target page and need no pool. All 200 targets are among the pages of commit 5abda21.

Ranker MRR Target first Target in top 10 Target in top 100
baseline (BM25F) 0.941 0.905 1.000 1.000
dense-minilm 0.790 0.695 0.945 1.000
hybrid-rrf 0.914 0.865 1.000 1.000

On these queries the dense ranker places the target lower: its reciprocal rank is lower by 0.151 on average ($t = -6.57$; ahead on 7 queries, behind on 55). The hybrid's MRR is below BM25F's by 0.027 ($t = -1.75$, $p = 0.082$; ahead on 12 queries, behind on 19), a difference these 200 queries do not distinguish from zero; its point estimate is lower. These repeat on the pages of commit 5abda21 the snapshot results of Dense Passage Retrieval (0.942, 0.795 and 0.916). The known-item queries were written from each page's title, summary and headings, which favors a lexical ranker that weights titles and summaries.

Overlap. On the topical queries the baseline and the dense ranker share 4.45 of their top ten pages on average, and for one query none; the Jaccard overlap of their top 20s is 0.336. The hybrid's top ten shares 6.65 pages with the baseline's and 6.72 with the dense ranker's.

The limits of the comparison. Each of these could change the result, and none was tested here.

  • The queries were written by a model, the topical ones from thread questions and course outlines and the known-item ones from the pages' summaries, and are not a sample of what readers type. The documentation expects them to favor lexical ranking, which, if so, works against the dense ranker on both sets and against the hybrid, which is half dense, to a lesser degree.
  • The grades come from a model judge. Its agreement with the owner was measured on 150 pairs of the first pool, and the batch that graded the new rankers' pairs was checked against a full-reading judge, not against the owner. Whether the two batches grade alike was not measured directly. Of the 775 pooled pairs outside BM25F's top 20, 676 were graded by the claude-sonnet-5-5-hybrid batch, so a difference in the batches' standards would bias the comparison in one direction.
  • The pool is three rankers deep at 20. A page that none of the three put in its top 20 is counted as not relevant. Since every top ten is judged, such a page could only raise a query's ideal DCG, lowering the three rankers' nDCG@10 for that query by the same factor: it cannot change which ranker is ahead on a query, but it can change the sizes of the differences. It also bounds R@20 to the pooled relevant pages.
  • Sixty queries leave intervals of several hundredths. The tune split alone does not separate the hybrid from BM25F.
  • The site's pages change. These numbers are for the 743 pages of commit 5abda21, with the pool, the grades and the query files as they stand at commit d4d95d6 (which changed only a date in the judging records). The script rebuilds those pages and reads those files from git, so later edits and later grades do not move them; evaluate.py reproduces the tables when run at commit 5abda21.

What the measurement does not decide

Whether to ship a dense stage is a decision for the owner, and effectiveness on these test sets is one input to it. The others are costs the measurements above do not weigh. The query must be encoded at search time, so the model must be loaded in the web process; its weights file alone (model.safetensors) is 90,868,376 bytes. The passage vectors take $13{,}031 \times 384 \times 4 = 20{,}015{,}616$ bytes for those pages at four bytes per number, and must be recomputed for every page that changes. The search box's instant results are computed in the reader's browser from precomputed term weights (How This Search Works); a dense or hybrid stage could not run there without shipping the model to the browser too, so the instant results and the full results would rank differently. This page reports the measurement. It does not make the decision.

What is proved here, what is cited, and what the experiment shows

Proved on this page. Proposition 1, that positive affine normalizations of the two scores produce the same family of rankings as the weight varies (the min-max and z-score case is Lemma 4.2 of Bruch, Gai and Ingber). Proposition 2, that a document beaten on both scores by $k$ others never enters the top $k$. Proposition 3, that the fused ranking changes at no more than $n(n-1)/2$ weights. Proposition 4, the recall ceiling of a fusion. The properties of RRF are cited from Rank Fusion, where they are proved.

Cited. The fusion methods, the conditions under which they help and their effects on TREC data, through Rank Fusion and its sources, of which Lee (1997) and Vogt and Cottrell (1998) are named here. Every published effectiveness number and conclusion about hybrids: Karpukhin et al.; Ma et al.; Wang, Zhuang and Zuccon; Chen et al.; Bruch, Gai and Ingber.

Shown by the experiment. On Cranfield, with one pretrained bi-encoder: the two rankers share fewer than half their top ten and about half as many of their nonrelevant abstracts as the lexical rankers do; every fusion tried beats both inputs, by about 0.05 in nDCG@10 over BM25; a cross-validated interpolation weight is not distinguishable from RRF; the per-query best weight varies so much that an oracle would add 0.07 more. On this site, with all three rankers pooled and every pooled pair graded by a model judge: the fusion of BM25F and the dense ranker is ahead of BM25F in nDCG@10 on the 60 topical queries (by 0.055, interval [0.023, 0.087]) and in R@20, the dense ranker alone is not distinguishable from BM25F, and on the known-item queries the fusion's MRR is not distinguishable from BM25F's and its point estimate is lower. Both test sets were written by a model, the known-item set from the pages' summaries. Nothing here decides whether the site should add a dense stage.

Worked problems

These problems are posed on small exact cases and on Cranfield query 185; the unit has no course assignments. The script recomputes each number, the exact ones in rational arithmetic.

Problem 1 (breakpoints). Four documents have min-max normalized scores (lexical, dense): $a$ (1.0, 0.2), $b$ (0.6, 0.7), $c$ (0.3, 1.0), $d$ (0.5, 0.4). Find the order under $f_\alpha$ for every $\alpha \in [0, 1]$.

Solution. The fused scores are lines: $f(a) = 1 - 0.8\alpha$, $f(b) = 0.6 + 0.1\alpha$, $f(c) = 0.3 + 0.7\alpha$, $f(d) = 0.5 - 0.1\alpha$. Setting pairs equal gives the crossings: $a$ and $b$ at $0.4/0.9 = 4/9$; $a$ and $c$ at $0.7/1.5 = 7/15$; $b$ and $c$ at $0.3/0.6 = 1/2$; $a$ and $d$ at $0.5/0.7 = 5/7$; $c$ and $d$ at $0.2/0.8 = 1/4$; $b$ and $d$ never, since $f(b) - f(d) = 0.1 + 0.2\alpha > 0$. In increasing order the crossings are $1/4, 4/9, 7/15, 1/2, 5/7$, so there are six intervals:

Weights Order
$0 \le \alpha < 1/4$ $a, b, d, c$
$1/4 < \alpha < 4/9$ $a, b, c, d$
$4/9 < \alpha < 7/15$ $b, a, c, d$
$7/15 < \alpha < 1/2$ $b, c, a, d$
$1/2 < \alpha < 5/7$ $c, b, a, d$
$5/7 < \alpha \le 1$ $c, b, d, a$

Five crossings for four documents, within the bound $4 \cdot 3/2 = 6$ of Proposition 3, since one pair never crosses. The first place passes from $a$ to $b$ to $c$; a tuner that tried only $\alpha = 0, 0.5, 1$ would never see $b$ first, which holds only on $(4/9, 1/2)$.

Problem 2 (the weight under another normalization). For query 185, BM25's scores over the 1,400 abstracts run from 0 to 18.038 and the dense cosines from −0.087 to 0.773. Min-max interpolation with $\alpha = 0.5$ ranks like which raw-score combination?

Solution. Min-max divides the lexical scores by their range $R_l$ and the dense scores by theirs, $R_s$; in Proposition 1's notation $a = 1/R_s$ and $c = 1/R_l$. Here $R_l = 18.038$ and $R_s = 0.773 - (-0.087) = 0.860$. With $\alpha = 0.5$, $A = 0.5/R_s$ and $B = 0.5/R_l$, so the equivalent raw weight is

$$\beta = \frac{A}{A + B} = \frac{1/R_s}{1/R_s + 1/R_l} = \frac{R_l}{R_l + R_s} = \frac{18.038}{18.038 + 0.860} = 0.954.$$

So an even split after normalization is a raw combination with almost all the weight on the cosine, because the cosines vary over a range 21 times smaller than BM25's. That is why the best raw weight on Cranfield was near 0.99. The conversion depends on the two ranges, which differ from query to query, so no single raw weight reproduces min-max interpolation on every query.

Problem 3 (which documents can reach the top two). In Problem 1, which documents does Proposition 2 allow into the top 2, and which actually get there?

Solution. Count, for each document, the documents better on both scores. $a$ (1.0, 0.2): none, since nothing has a higher lexical score. $b$ (0.6, 0.7): only $a$ has a higher lexical score, and its dense score is lower; none. $c$ (0.3, 1.0): nothing has a higher dense score; none. $d$ (0.5, 0.4): $b$ beats it on both; one. All four have fewer than two, so all four are allowed. From the table of Problem 1, $a$, $b$ and $c$ each reach the top two and $d$ never does: it is third at best. Proposition 2 is a necessary condition only.

Problem 4 (RRF by hand on query 185). Abstract 856 is ranked 1st by BM25 and 3rd by the dense retriever; abstract 658 is 7th and 1st; abstract 857 is 2nd and 6th; abstract 864 is 8th and 2nd. Order them by RRF with $k_{\mathrm{RRF}} = 60$.

Solution.

$$\mathrm{RRF}(856) = \tfrac{1}{61} + \tfrac{1}{63} = 0.03227, \quad \mathrm{RRF}(658) = \tfrac{1}{67} + \tfrac{1}{61} = 0.03132,$$ $$\mathrm{RRF}(857) = \tfrac{1}{62} + \tfrac{1}{66} = 0.03128, \quad \mathrm{RRF}(864) = \tfrac{1}{68} + \tfrac{1}{62} = 0.03083.$$

The order is 856, 658, 857, 864, the first four places of the fused list. Abstract 658, the dense retriever's first choice and not judged relevant, comes second because both rankers put it in their top ten. Abstract 766, relevant and 3rd for BM25, is 11th in the dense list ($\tfrac{1}{63} + \tfrac{1}{71} = 0.02996$) and falls to 7th. On this query, agreement promoted a wrong abstract and demoted a right one. That is part of why the fused list scores 0.521 against BM25's 0.807; the rest is that the relevant abstracts 858 and 859, 5th and 9th for BM25, leave the fused top ten, and the nonrelevant 864, 948 and 627 enter it, so the fused top ten holds 4 relevant abstracts against BM25's 6.

Problem 5 (the recall ceiling). For one query with 10 relevant abstracts, BM25's top 100 holds 7 of them and the dense top 100 holds 6, with 4 in both. What is the most recall@100 any fusion of the two lists can reach, and what is the most recall@10?

Solution. The union holds $7 + 6 - 4 = 9$ relevant abstracts, so by Proposition 4 recall@100 is at most $9/10 = 0.9$, and the union has at most 200 documents, of which the fusion keeps 100: the 9 fit, so 0.9 is reachable. Recall@10 is at most $9/10$ by the same proposition, and reaching it would need nine of the ten fused places to be relevant. Without the dense list the ceiling would be $7/10$; adding a list raised it by the two relevant abstracts only that list found. On Cranfield as a whole the union of the two top 100s holds 0.834 of the relevant abstracts, against 0.737 for BM25's alone.

What we have learned

A lexical ranker and a dense retriever err differently enough on Cranfield for fusion to work: they share most relevant abstracts and only about half as many wrong ones as lexical rankers share with each other, and every fusion tried improves on both inputs by about 0.05 in nDCG@10. Rank fusion needs no normalization. Interpolation needs a weight, and the propositions here make it exactly tunable: normalization only rescales the weight, a document beaten on both scores cannot rise, and the ranking is piecewise constant in the weight. Tuned with cross-validation, interpolation was not distinguishable from RRF on this collection; the per-query best weight varies widely, and an oracle choosing it would gain far more than any fixed weight. On this site, with every ranker's top 20 judged, the same fusion is ahead of BM25F on the topical queries and not distinguishable from it on the known-item queries, on test sets written by a model; whether to add a dense stage is the owner's decision, which also weighs the model's size, memory, encoding cost and the browser's instant results.

Fusion combines two finished lists. The other way to get lexical precision and learned matching together is a single representation that has both: Learned Sparse Retrieval keeps the vocabulary's dimensions and learns their weights, and Late Interaction and ColBERT keeps a vector per token. The fused list still has to be ordered precisely at the top, which is the work of cross-encoder re-ranking, and it may hold several near-identical passages, which Result Diversification addresses.

References