Cranfield query 185 asks for experimental studies on panel flutter. Three rankers of this unit answer it differently. BM25 and query likelihood both put abstract 856 first. TF-IDF with cosine normalization puts abstract 864 first, a 35-token abstract that the other two rank 8th and 11th. Comparing ranking models treated such rankers as rivals and picked a winner. Can their three lists instead be merged into one list that is better than any of them?
They can always be merged, and the merged list is sometimes better. Rank fusion (also called data fusion or, when the inputs are whole search engines, metasearch) takes several ranked lists for the same query and produces one. The published experiments this article cites found fusion better than the best input on TREC collections. On Cranfield, with this unit's three rankers, it is not: every fused list scores between the best input and the worst. Both results are reported here, with the reasons the literature gives for when fusion helps.
Here is the route. First we see why the scores of different rankers cannot simply be added. Then we define the score-based methods, CombSUM and CombMNZ, with the normalization they need, and the rank-based methods: the Borda count, reciprocal rank fusion and Condorcet fusion. Next come the reasons fusion can help, and the line between what is provable about these methods and what is only measured. Finally we fuse the unit's rankers on all 225 Cranfield queries. The article assumes comparing ranking models and the measures of retrieval evaluation. The numbers are computed by tools/experiments/cranfield/rank_fusion.py, which is deterministic apart from a seeded randomization test.
Scores that cannot be added
Each ranker gives each abstract a score, and the obvious fusion is to add the scores. For query 185 the three score scales are these:
| Ranker | Lowest score among the 542 abstracts retrieved | Highest score | Score of abstract 856 | Score of abstract 390 |
|---|---|---|---|---|
| BM25 | 0.767 | 18.038 | 18.038 | 14.681 |
| Query likelihood (Dirichlet, $\mu = 200$) | −3.092 | 9.373 | 9.373 | 5.745 |
| TF-IDF (cosine) | 0.017 | 0.369 | 0.326 | 0.297 |
The scales have nothing in common. BM25's scores are sums of weighted term contributions with no upper limit. The query likelihood scores are the rank-equivalent log probabilities of the language-model article, and many are negative. The cosine scores lie between 0 and 1. A plain sum is dominated by whichever ranker has the widest scale: the whole range of the TF-IDF scores, 0.35, is smaller than the gap between the first and second BM25 scores. The Probability Ranking Principle article made the same point from the other side: these scores order documents for one query and are not calibrated probabilities, so their sizes carry no shared meaning.
There are two ways out. One makes the scores comparable before adding them. The other throws the scores away and uses only the ranks.
Score-based fusion
Fox and Shaw (1994) defined the standard family of score-based methods in their TREC-2 experiments. For each document they combined its similarity values from the individual runs in one of six ways:
| Method | Fused score of a document |
|---|---|
| CombMAX | the largest of its scores |
| CombMIN | the smallest of its scores |
| CombSUM | the sum of its scores |
| CombANZ | the sum divided by the number of runs that retrieved it |
| CombMNZ | the sum multiplied by the number of runs that retrieved it |
| CombMED | the median of its scores |
A run that did not retrieve the document contributes a score of zero. Two of the six became standard. Write $s_i(d)$ for the score of document $d$ in run $i$, with $s_i(d) = 0$ if run $i$ did not retrieve $d$, and $m(d)$ for the number of runs that retrieved it. Then
$$\mathrm{CombSUM}(d) = \sum_i s_i(d), \qquad \mathrm{CombMNZ}(d) = m(d) \cdot \sum_i s_i(d).$$
CombMNZ ("multiply by the number of nonzero scores") was designed, in Fox and Shaw's words, "to provide higher weights to documents retrieved by multiple retrieval methods". They found CombMIN and CombMAX too specialized, each guarding against one kind of error at the cost of the other, and reported that CombSUM was significantly better than their best single run.
Normalization
Fox and Shaw combined five runs of their own system and, as Lee (1997) notes, applied no normalization. Lee combined the runs of six different TREC-3 systems, observed that "different retrieval results can generate quite different ranges of similarity values", and normalized each run before combining it. His normalization rescales a run's scores for a query so that the lowest becomes 0 and the highest 1:
$$s'_i(d) = \frac{s_i(d) - \min_i}{\max_i - \min_i},$$
where $\min_i$ and $\max_i$ are the smallest and largest scores run $i$ gave to the documents it retrieved for that query. This min-max normalization is what "CombSUM" and "CombMNZ" mean in most later work, including this article unless "raw" is stated. With it, Lee found CombMNZ slightly better than CombSUM on his runs.
Back to query 185. For abstract 856 the three normalized scores are 1.000, 1.000 and 0.876, because it has the highest score in the BM25 and query likelihood runs. For TF-IDF the normalized score is $(0.326 - 0.017)/(0.369 - 0.017)$, which is 0.876 from the unrounded scores. Its CombSUM score is 2.876, the highest of any abstract. For abstract 390 they are 0.806, 0.709 and 0.793, a sum of 2.308, which places it fourth. All three runs retrieve the same 542 abstracts for this query, so $m(d) = 3$ for every one of them. CombMNZ then multiplies every score by 3 and gives the same ranking as CombSUM, a fact proved in general below.
Normalization removes the difference in scale. It does not make the scores mean the same thing: a normalized 0.8 in one run and in another say only that each document is four fifths of the way from that run's worst score to its best.
Rank-based fusion
The second way out uses only each document's position in each list. Rank-based methods need no scores, which also makes them usable when a search engine reports none.
The Borda count
The Borda count is a voting rule from the eighteenth century, applied to retrieval by treating each ranker as a voter and each document as a candidate. In the description of Montague and Aslam (2002), with $n$ candidates each voter gives its top candidate $n$ points, the second $n - 1$, and so on, and the candidate with the most points wins. A document's fused score is its total of points, so the method ranks documents by their average rank. Two conventions are needed in practice and are used in this article's script. The candidates are the documents that at least one run retrieved. A run's remaining points are shared equally among the documents it did not retrieve.
Reciprocal rank fusion
Cormack, Clarke and Büttcher (2009) proposed a simpler rule, reciprocal rank fusion (RRF). If $r_i(d)$ is the rank of document $d$ in run $i$, starting at 1, then
$$\mathrm{RRF}(d) = \sum_i \frac{1}{k + r_i(d)},$$
where the sum runs over the runs that retrieved $d$, and $k$ is a constant. (Cormack, Clarke and Büttcher take every run to rank every document; leaving out the runs that did not retrieve $d$ is the usual convention for cut lists, and it is the one used here.) The authors set $k = 60$, a value that in their words "was fixed during a pilot investigation and not altered during subsequent validation". Their pilot table shows mean average precision varying only in the third decimal place for $k$ between 20 and 100, and they describe the choice as "not critical".
Here is the idea behind the formula. A reciprocal makes high ranks count for more than low ones, but with $k = 0$ it would count them for far too much: rank 1 would be worth ten times rank 10. Adding 60 flattens the curve. Rank 1 is then worth $1/61 = 0.01639$ and rank 10 is worth $1/70 = 0.01429$, a ratio of 1.15. The authors give this as the purpose of the constant: it "mitigates the impact of high rankings by outlier systems".
For query 185, abstract 856 is ranked 1st, 1st and 3rd by BM25, query likelihood and TF-IDF:
$$\mathrm{RRF}(856) = \frac{1}{61} + \frac{1}{61} + \frac{1}{63} = 0.04866.$$
Abstract 864, TF-IDF's first choice, is ranked 8th, 11th and 1st: $\frac{1}{68} + \frac{1}{71} + \frac{1}{61} = 0.04518$. One first place does not outweigh two modest ranks, and abstract 864 is seventh in the fused list. The fused top seven are abstracts 856, 857, 1008, 766, 390, 858 and 864, of which all but 390 and 864 are relevant.
RRF is used in production search systems to merge a lexical ranking, such as BM25's, with a ranking from vector search: Azure AI Search documents it as the scoring method for its hybrid queries (Microsoft, 2026). A later unit covers that kind of retrieval.
Condorcet fusion
The Borda count and RRF are positional methods: each run awards a document a number that depends on its position, and the numbers are added. Montague and Aslam (2002) built a fusion method on the other classical school of voting, the majoritarian one. In it, document $d_1$ is placed above $d_2$ if more of the input rankings put $d_1$ above $d_2$ than the reverse. Pairwise majorities need not be consistent. With the three rankings $a, b, c$ and $b, c, a$ and $c, a, b$, a majority prefers $a$ to $b$, $b$ to $c$ and $c$ to $a$, so no ordering satisfies them all; this is the voting paradox. Montague and Aslam treat the documents on such a cycle as tied, and show that a consistent ordering of the remaining structure can be found by sorting the documents with the pairwise majority vote as the comparison, in $O(nk \log n)$ time for $n$ documents and $k$ rankings. They call the algorithm Condorcet-fuse and report that on TREC data it significantly outperformed Borda-fuse. Cormack, Clarke and Büttcher (2009) later reported RRF ahead of Condorcet-fuse in each of their comparisons.
Why fusion can help
Fusion is not guaranteed to help, and the literature offers three mechanisms by which it can. Vogt and Cottrell (1998) list them in the words of Ted Diamond's 1996 PhD dissertation proposal, which they cite:
- The skimming effect happens when "retrieval approaches that represent their collection items differently may retrieve different relevant items, so that a combination method that takes the top-ranked items from each of the retrieval approaches will push non-relevant items down in the ranking."
- The chorus effect occurs "when several retrieval approaches suggest that an item is relevant to a query", which "tends to be stronger evidence for relevance than a single approach doing so."
- The dark horse effect is that "a retrieval approach may produce unusually accurate (or inaccurate) estimates of relevance for at least some items, relative to the other retrieval approaches."
The three pull against each other. Vogt and Cottrell note that the dark horse effect "is at odds with the Chorus Effect", since trusting an outlier is the opposite of trusting agreement, and that a large chorus effect cuts into what skimming can gain. Each fusion method is a bet on one of them. CombMNZ and RRF with a large $k$ bet on the chorus: agreement across runs is rewarded. CombMAX, which keeps only a document's best score, can be read as a bet on the dark horse.
Lee (1997) gave the chorus effect a measurable form. On TREC-3 runs he found that different runs "retrieve similar sets of relevant documents but retrieve different sets of nonrelevant documents", which he called the unequal overlap property. He measured it for two runs by the overlap coefficient $2c/(n_1 + n_2)$, where $c$ is the number of relevant documents the runs have in common and $n_1$ and $n_2$ the numbers of relevant documents each retrieved, and by the same coefficient for nonrelevant documents. If the property holds, a document retrieved by many runs is more likely relevant, which is what CombMNZ rewards. Vogt and Cottrell reached a matching conclusion for weighted sums of scores: the systems combined should perform well individually and have a high overlap of relevant documents and a low overlap of nonrelevant ones.
What is provable
Some properties of these methods follow from their definitions and need no experiment. Four are proved here. Everything about effectiveness is in the next section, and is empirical.
1. Rank-based methods are unaffected by any strictly increasing rescaling of an input's scores. Let run $i$ rank documents by scores $s_i(d)$, and let $g$ be a strictly increasing function. Replace the scores by $g(s_i(d))$. For any two documents, $s_i(d) > s_i(d')$ exactly when $g(s_i(d)) > g(s_i(d'))$, and equal scores stay equal, so every document has the same rank $r_i(d)$ as before. The Borda count, RRF and Condorcet fusion are functions of the ranks alone, so their output is unchanged. In particular the exponential, any change of units, and, for positive scores, the logarithm and any positive power leave these methods' results exactly as they were. That is why RRF can combine BM25 scores, log probabilities and cosines without asking what any of them mean.
2. CombSUM is not. A counterexample is enough. Three runs score three documents:
| Run | $x$ | $y$ | $z$ | Ranking |
|---|---|---|---|---|
| A | 10 | 9 | 0 | $x, y, z$ |
| B | 10 | 9 | 0 | $x, y, z$ |
| C | 1 | 10 | 0 | $y, x, z$ |
After min-max normalization the scores are $(1, 0.9, 0)$ in A and in B and $(0.1, 1, 0)$ in C. CombSUM gives $x$ a total of $1 + 1 + 0.1 = 2.1$ and $y$ a total of $0.9 + 0.9 + 1 = 2.8$, so it ranks $y$ first. Now rescale run C by the strictly increasing function $g(s) = s^{0.05}$. Its scores become $1^{0.05} = 1$ for $x$, $10^{0.05} = 1.122$ for $y$ and 0 for $z$, and its ranking is still $y, x, z$. Normalized, they are $(0.891, 1, 0)$. CombSUM now gives $x$ a total of $2.891$ and $y$ still $2.8$, and ranks $x$ first. The same three rankings produced two different fused orders, according to how one run happened to space its scores. RRF gives $x$ the score $\tfrac{2}{61} + \tfrac{1}{62} = 0.04892$ and $y$ the score $\tfrac{2}{62} + \tfrac{1}{61} = 0.04865$ both times, and ranks $x$ first, with the two runs that agree. Raw CombSUM is more fragile still: with the original scores it ranks $y$ first, 28 to 21, and dividing run C's scores by 100 makes it rank $x$ first, 20.01 to 18.1.
The script repeats the comparison on 2,000 random cases of three runs over eight documents, each run rescaled by a random power. The RRF and Borda rankings never change, as the proof requires. The min-max CombSUM ranking changes in 1,981 of the 2,000 cases. So the dependence on score spacing is the normal case for CombSUM and not a curiosity of one example. It can also be an advantage: if a run's score gaps carry real information about relevance, CombSUM uses it and RRF discards it.
3. In RRF, agreement outweighs one first place up to rank $k + 1$. Take two runs. Document $u$ is ranked $r$ in both, and document $v$ is ranked first in one run and not retrieved by the other. Then $\mathrm{RRF}(u) = \frac{2}{k + r}$ and $\mathrm{RRF}(v) = \frac{1}{k + 1}$, and
$$\frac{2}{k + r} > \frac{1}{k + 1} \iff 2(k + 1) > k + r \iff r < k + 2.$$
With $k = 60$, a document that both runs rank anywhere from 2nd to 61st beats a document that one run ranks first and the other misses (rank 1 is taken by $v$ in that run). With $k = 0$ agreement never wins: $r = 1$ is impossible for the same reason, and at $r = 2$ the two scores tie, $2/2 = 1/1$. The constant $k$ therefore sets how deep agreement reaches: it is the dial between the chorus effect and the dark horse effect. The same calculation for $m$ runs gives $r < m(k + 1) - k$.
4. CombMNZ ranks as CombSUM does wherever every run retrieves every document. If $m(d) = m$ for all the documents compared, then $\mathrm{CombMNZ}(d) = m \cdot \mathrm{CombSUM}(d)$, and multiplying all scores by the same positive constant changes no order. The two methods can differ only through documents that some run failed to retrieve. Fox and Shaw made the same observation for CombANZ and CombMNZ, which, they noted, produce the same ranked sequence for the documents retrieved by all five of their runs. The argument above shows that CombSUM joins them. For query 185 all three of our runs retrieve the same 542 abstracts, which is why the two methods agreed there.
What is empirical: fusion on Cranfield
Whether fusion improves a ranking is a question about data. The published answers are favorable. Fox and Shaw (1994) found CombSUM significantly better than their best single run. Lee (1997) found improvements from combining TREC-3 runs, with CombMNZ best. Cormack, Clarke and Büttcher (2009) report that RRF outperformed Condorcet-fuse, CombMNZ and the best individual system "by 4% to 5% on average" over their pilot and TREC experiments. Those experiments combined runs from different systems and research groups.
This unit's rankers are a harder case, and the experiment below was run to see what happens with them. The inputs are three runs on all 225 Cranfield queries, each with the text processing of comparing ranking models and cut at 1,000 documents:
- BM25 with $k_1 = 1.2$ and $b = 0.75$;
- query likelihood with Dirichlet smoothing, $\mu = 200$, the value the development queries chose in language models for retrieval;
- TF-IDF with lnc.ltc weighting and cosine normalization, the vector space run.
The measures are mean average precision (MAP) over the top 1,000 and nDCG at 10, with two-sided paired randomization tests against BM25: the 225 per-query differences in average precision have their signs flipped at random 20,000 times (seed 1), and $p$ is the share of flips whose mean difference is at least as large in absolute value as the observed one. The rankers are deterministic, so there are no repeated runs to summarize. Because $\mu$ was tuned on half the queries, the last two columns repeat the measures on the other half, the 113 test queries of the unit's seeded split.
| Run | MAP | nDCG@10 | MAP, test half | nDCG@10, test half |
|---|---|---|---|---|
| BM25 | 0.307 | 0.368 | 0.297 | 0.347 |
| TF-IDF | 0.294 | 0.356 | 0.289 | 0.341 |
| Query likelihood | 0.284 | 0.350 | 0.284 | 0.338 |
| CombSUM, raw scores | 0.301 | 0.364 | 0.295 | 0.345 |
| CombMNZ, raw scores | 0.301 | 0.364 | 0.295 | 0.345 |
| CombSUM, min-max | 0.299 | 0.362 | 0.293 | 0.348 |
| CombMNZ, min-max | 0.299 | 0.362 | 0.293 | 0.348 |
| Borda count | 0.296 | 0.360 | 0.288 | 0.343 |
| RRF, $k = 60$ | 0.298 | 0.362 | 0.290 | 0.344 |
Several things can be read from the table.
- No fusion of the three beats BM25. Every fused run has a lower MAP than BM25 over all 225 queries and a higher MAP than query likelihood, the weakest input. RRF is 0.008 below BM25, better on 85 queries and worse on 122 ($p = 0.008$). The Borda count is 0.011 below ($p = 0.002$). For min-max CombSUM the difference of 0.008 is not significant ($p = 0.081$). On the test half, nDCG@10 for min-max CombSUM is 0.348 against BM25's 0.347, a difference of no consequence.
- The methods barely differ from each other. The fused MAPs span 0.296 to 0.301. CombMNZ equals CombSUM to three decimals. On 102 of the 225 queries the three runs retrieve exactly the same abstracts, and property 4 makes the two fused lists identical. On the other 123, more than 1,000 abstracts contain a query term, the cut lists differ, and so do the two fused orders, but not enough to move MAP or nDCG@10 in the third decimal.
- Raw CombSUM looks best for a bad reason. With unnormalized scores the sum is dominated by BM25, whose scale is the widest, so the fused list is close to BM25's own. It is the least fused of the fusions.
- The constant $k$ hardly matters here. For RRF over the three runs, MAP is 0.301 at $k = 0$, 0.302 at $k = 1$, 0.300 at $k = 10$, 0.298 at $k = 60$ and 0.299 at $k = 500$. This agrees with the authors' remark that the choice is not critical, though on this collection the smallest values are marginally the best.
Leaving out the weakest input helps a little and changes no conclusion. RRF of BM25 and TF-IDF alone reaches a MAP of 0.300, and min-max CombSUM of the same two reaches 0.306 (better than BM25 on 101 queries, worse on 109, $p = 0.97$), which on the test half is 0.302 against BM25's 0.297. That is the only comparison in which a fused run has a higher MAP than BM25, and the test on all 225 queries gives no support for a real difference.
Why fusion does not help here
The literature's conditions explain the result, at least in part. The script measures Lee's overlap on the top 100 documents of each pair of runs, summed over the queries:
| Pair of runs | Overlap of relevant documents | Overlap of nonrelevant documents |
|---|---|---|
| BM25 and query likelihood | 0.930 | 0.722 |
| BM25 and TF-IDF | 0.965 | 0.867 |
| Query likelihood and TF-IDF | 0.951 | 0.803 |
The relevant overlap is high, as fusion wants. But the nonrelevant overlap is high too, between 0.72 and 0.87. The three rankers are built on the same evidence: the same tokens, the same stemmer, the same term counts, document frequencies and lengths. They make largely the same mistakes, so a document's presence in several lists says little that one list did not already say, and the chorus has one voice. A ranker that draws on different evidence, such as the term proximity of term dependence models, would be a more promising partner; that fusion was not run. There is also little to skim: before the cut at 1,000, each ranker retrieves every abstract containing a query term, so no ranker finds relevant abstracts that the others cannot. What remains is averaging a better ranker with two worse ones. This is an explanation consistent with the measurements and with Lee's and Vogt and Cottrell's conditions. The experiment does not test it against alternatives.
Example (a common misstatement). "Fusing rankings always improves on the best of them." The table above is a counterexample: three rankers, six fused runs, and none above BM25. (Condorcet fusion was not run.) The precise statement is that fusion has improved on the best input in published experiments where the inputs were effective and differed in the nonrelevant documents they retrieved, and that the methods themselves guarantee nothing about effectiveness. On query 185 alone, RRF does come out ahead, with an average precision of 0.791 against 0.784 for BM25, 0.774 for query likelihood and 0.624 for TF-IDF. That is one query, and min-max CombSUM on the same query scores 0.761.
Worked problems
The course text does not treat fusion, so Problems 1 to 3 are posed on small constructed lists and Problem 4 on query 185. The script recomputes every number.
Problem 1. Two runs score five documents. Run 1 gives $a$ 12, $b$ 9, $c$ 6 and $d$ 3, and does not retrieve $e$. Run 2 gives $c$ 0.9, $a$ 0.5, $e$ 0.4 and $b$ 0.1, and does not retrieve $d$. Fuse them by min-max CombSUM, CombMNZ, RRF with $k = 60$, and the Borda count.
Normalization. Run 1 has minimum 3 and maximum 12, so its normalized scores are $a$ 1, $b$ $6/9 = 0.667$, $c$ $3/9 = 0.333$, $d$ 0. Run 2 has minimum 0.1 and maximum 0.9: $c$ 1, $a$ $0.4/0.8 = 0.5$, $e$ $0.3/0.8 = 0.375$, $b$ 0.
| Document | Run 1 rank, normalized score | Run 2 rank, normalized score | CombSUM | CombMNZ | RRF | Borda |
|---|---|---|---|---|---|---|
| $a$ | 1, 1.000 | 2, 0.500 | 1.500 | 3.000 | 0.03252 | 9 |
| $b$ | 2, 0.667 | 4, 0.000 | 0.667 | 1.333 | 0.03175 | 6 |
| $c$ | 3, 0.333 | 1, 1.000 | 1.333 | 2.667 | 0.03227 | 8 |
| $d$ | 4, 0.000 | not retrieved | 0.000 | 0.000 | 0.01563 | 3 |
| $e$ | not retrieved | 3, 0.375 | 0.375 | 0.375 | 0.01587 | 4 |
CombMNZ multiplies the sum by 2 for $a$, $b$ and $c$, which both runs retrieved, and by 1 for $d$ and $e$. RRF: for $a$, $\tfrac{1}{61} + \tfrac{1}{62} = 0.03252$; for $c$, $\tfrac{1}{63} + \tfrac{1}{61} = 0.03227$; for $d$, only $\tfrac{1}{64} = 0.015625$. Borda: the pool has $n = 5$ documents, so a run gives 5, 4, 3 and 2 points to its four documents, and its one remaining point to the document it missed. For $a$ that is $5 + 4 = 9$, and for $e$ it is $1 + 3 = 4$.
All four methods give the same order: $a, c, b, e, d$. Document $a$ wins because it is near the top of both lists, and $c$, first in one list and third in the other, is second. The two documents retrieved by only one run come last under every method.
Problem 2. With two runs and RRF, document $u$ is ranked 40th by both runs. Document $v$ is ranked 1st by one run and is not retrieved by the other. Which is ranked higher with $k = 60$, and which with $k = 0$? What is the lowest shared rank at which $u$ still wins with $k = 60$?
With $k = 60$: $\mathrm{RRF}(u) = 2/100 = 0.0200$ and $\mathrm{RRF}(v) = 1/61 = 0.0164$, so $u$ is higher. With $k = 0$: $\mathrm{RRF}(u) = 2/40 = 0.05$ and $\mathrm{RRF}(v) = 1$, so $v$ is higher, by a factor of 20. By property 3, $u$ wins exactly when $r < k + 2$ (and $r \ge 2$, since $v$ holds rank 1 in one of the runs). For $k = 60$ the lowest winning rank is 61: at $r = 61$, $2/121 > 1/61$; at $r = 62$ the two scores are equal, $2/122 = 1/61$; at $r = 63$, $v$ wins. With three runs agreeing on rank $r$ against one first place, the condition is $r < 3 \times 61 - 60 = 123$.
Problem 3. Show by example that CombMNZ and CombSUM can order two documents differently, and state when they cannot.
Take two min-max normalized runs. Run 1 scores $u$ 1.0, $v$ 0.45 and $w$ 0. Run 2 scores $x$ 1.0, $v$ 0.45 and $y$ 0, and does not retrieve $u$. CombSUM gives $u$ a total of 1.0 and $v$ a total of 0.9, and ranks $u$ first. CombMNZ gives $u$ a score of $1 \times 1.0 = 1.0$ and $v$ a score of $2 \times 0.9 = 1.8$, and ranks $v$ first. The methods differ because the documents were retrieved by different numbers of runs. By property 4 they cannot differ on any pair of documents retrieved by the same number of runs, since both scores are then multiplied by the same $m$. CombMNZ is CombSUM with an explicit reward for the chorus effect.
Problem 4. For query 185, abstract 766 is ranked 3rd, 3rd and 9th by BM25, query likelihood and TF-IDF, and abstract 1008 is ranked 6th, 6th and 2nd. Two of the three rankers prefer 766. Compute both RRF scores and explain the fused order.
$$\mathrm{RRF}(766) = \frac{2}{63} + \frac{1}{69} = 0.04624, \qquad \mathrm{RRF}(1008) = \frac{2}{66} + \frac{1}{62} = 0.04643.$$
Abstract 1008 is ranked higher, third in the fused list against fourth for 766. Compare the two differences. Moving from rank 3 to rank 6 in two runs costs $2\left(\tfrac{1}{63} - \tfrac{1}{66}\right) = 0.00144$. Moving from rank 9 to rank 2 in one run gains $\tfrac{1}{62} - \tfrac{1}{69} = 0.00164$. With $k = 60$ the curve $1/(k + r)$ is nearly straight over the top ranks, so RRF behaves there almost like an average of ranks, and 1008 has the smaller rank sum, $6 + 6 + 2 = 14$ against $3 + 3 + 9 = 15$. A majority vote between the two would choose 766, since two of the three rankers place it higher. Condorcet fusion is built on such votes; its order for a pair also depends on whether a cycle of majorities passes through both documents, which this article's script does not compute. Both abstracts are relevant, so here the difference costs nothing.
What we have learned
Several rankings of the same documents can be merged by their scores or by their ranks. Scores from different rankers are on unrelated scales, so CombSUM and CombMNZ need normalization, and even then their result depends on how each ranker spaces its scores. Rank-based methods, the Borda count, reciprocal rank fusion and Condorcet fusion, are provably unaffected by any strictly increasing rescaling. In RRF the constant $k = 60$ flattens the reward for high ranks, so that agreement between rankers outweighs one ranker's first place down to rank 61. Fusion can help through the skimming, chorus and dark horse effects, and published experiments on TREC runs found that it did. On Cranfield, fusing BM25, query likelihood and TF-IDF did not beat BM25 alone in any of the six fused runs (Condorcet fusion was not run), and the high overlap of the three rankers' nonrelevant documents is consistent with the conditions the literature gives.
That closes the unit on probabilistic retrieval. Every model in it ranks documents for a query that someone has just typed. The next unit treats the case in which the same question is asked of every new document as it arrives, such as whether it is about panel flutter. That is classification, and the text classification problem begins it.
References
- Cormack, G. V., Clarke, C. L. A., and Büttcher, S. (2009). Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proceedings of SIGIR 2009, 758–759. record
- Fox, E. A., and Shaw, J. A. (1994). Combination of multiple searches. In The Second Text REtrieval Conference (TREC-2), NIST Special Publication 500-215, 243–252. record
- Lee, J. H. (1997). Analyses of multiple evidence combination. In Proceedings of SIGIR 1997, 267–276. record
- Microsoft (2026). Hybrid search scoring (RRF). Azure AI Search documentation. https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking record
- Montague, M., and Aslam, J. A. (2002). Condorcet fusion for improved retrieval. In Proceedings of CIKM 2002, 538–548. record
- Vogt, C. C., and Cottrell, G. W. (1998). Predicting the performance of linearly combined IR systems. In Proceedings of SIGIR 1998, 190–196. record