The course presents four models of retrieval in sequence: Boolean retrieval, the vector space model with TF-IDF weighting, the Binary Independence Model, and BM25. Ranking was introduced to overcome exact matching, and BM25 to add term frequency and length to the BIM. This article tests those motivations directly, by running all four on the same test collection with the same text processing and measuring them with the methods of evaluation (Manning, Raghavan, and Schütze, 2008, chapters 6, 8, and 11). The experiment is small, but it is a real one, and its results agree with what the field found at much larger scale.
Setup
Collection. The Cranfield collection, from the experiments that founded test-collection evaluation (Cleverdon, 1967): 1,400 abstracts of aerodynamics papers and 225 queries, each a natural-language question written by a researcher, with graded relevance judgments (see test collections), loaded with the ir_datasets library (MacAvaney et al., 2021). Grades 1 to 4 were treated as relevant for binary measures. Grade 1 marks a complete answer and grade 4 a reference of minimum interest, so nDCG used 5 minus the grade as the gain. Queries have 7.2 relevant documents on average.
Text processing. Every model used the same analysis: lowercase, split on non-alphanumeric characters, remove a 33-word stop list (Lucene's English list), and stem with the original Porter algorithm. The documents average 111.6 tokens after processing, and the vocabulary has 4,803 terms.
Models.
| Model | Scoring |
|---|---|
| Boolean AND | documents containing every query term, in document order |
| TF-IDF | cosine similarity with lnc.ltc weighting (see variants of TF-IDF) |
| BIM | Robertson–Spärck Jones weight with no relevance information: $p_t = 0.5$, smoothed $u_t$, as in my HW04 |
| BM25 | $k_1 = 1.2$, $b = 0.75$, Lucene's IDF (see implementing BM25) |
Measures. Mean average precision over the top 1,000 results (the feedback and variant experiments in other articles rank all 1,400 documents, which makes no practical difference here), precision at 10, and nDCG at 10, with paired randomization tests on per-query average precision (see significance testing).
Results
| Model | MAP | P@10 | nDCG@10 | Queries with no results |
|---|---|---|---|---|
| Boolean AND | 0.010 | 0.006 | 0.013 | 216 |
| BIM (no relevance information) | 0.220 | 0.172 | 0.268 | 0 |
| TF-IDF (lnc.ltc) | 0.294 | 0.228 | 0.356 | 0 |
| BM25 | 0.307 | 0.234 | 0.368 | 0 |
Pairwise comparisons on average precision:
| Comparison | MAP difference | Queries better | Queries worse | $p$ |
|---|---|---|---|---|
| TF-IDF over BIM | +0.074 | 166 | 51 | < 0.001 |
| BM25 over BIM | +0.086 | 192 | 25 | < 0.001 |
| BM25 over TF-IDF | +0.012 | 129 | 84 | 0.075 |
MAP condenses each model's performance into one number. The interpolated precision-recall curves, averaged over the 225 queries, show where the differences lie (see evaluating ranked retrieval):
What the results show
Boolean AND fails on natural-language queries. The Cranfield queries are full questions, such as "what similarity laws must be obeyed when constructing aeroelastic models of heated high speed aircraft". After stop-word removal and stemming they still have 11.6 distinct terms on average (from 3 to 29). Requiring every one of those words to appear in a document returns nothing for 216 of the 225 queries. The nine queries that do return something are among the shortest, with four to seven terms, and between them they retrieve 20 documents. This is the feast-or-famine problem of the Boolean model, in its famine form. Boolean retrieval was designed for short queries carefully composed by expert searchers, not for questions in plain language. On such questions, ranked retrieval returns useful results where strict conjunction returns none.
Term frequency matters. The BIM sees only whether a term is present. TF-IDF and BM25 also count how often it occurs and correct for document length. Both beat the BIM decisively, by a third or more in MAP, and on most queries individually. This reproduces the finding, from the early TREC evaluations, that binary probabilistic weighting fell short of weighting that used term frequency, which was the motivation for BM25 (Robertson and Walker, 1994).
BM25 and TF-IDF are close. BM25 is better on average and wins on more queries, 129 to 84, but the difference in MAP, 0.012, is not significant at the 0.05 level. On a small collection of short, uniform abstracts, carefully normalized TF-IDF is a strong method, and BM25's advantages, saturation and tunable length normalization, have less to work with. On larger and more varied collections, BM25's margin is typically larger and more consistent.
Per-query differences
A difference of 0.012 in MAP is an average over queries that differ far more individually. Plotting each query's average precision under the two models shows how the average arises:
Most queries sit close to the diagonal: the two models usually rank the same documents near the top. The average is decided by a few dozen queries on which they disagree sharply, in both directions.
One query, scored by each model
Query 17 is BM25's largest loss against TF-IDF. It asks "can the three-dimensional problem of a transverse potential flow about a body of revolution be reduced to a two-dimensional problem", which analyzes to 14 terms, among them dimension and problem twice each. It has two relevant documents, 106 and 196. TF-IDF ranks document 106 first and has an average precision of 0.514; the other model ranks it sixth and has 0.092. The contributions of each query term to document 106 and to that model's first-ranked document, 1108 (not relevant), show why:
| Term | df | Doc 106 tf | Doc 1108 tf | TF-IDF, 106 | TF-IDF, 1108 | BM25, 106 | BM25, 1108 |
|---|---|---|---|---|---|---|---|
| transvers | 61 | 2 | 0.086 | 5.18 | |||
| potenti | 58 | 3 | 0.099 | 5.74 | |||
| revolut | 77 | 3 | 2 | 0.090 | 0.043 | 5.23 | 3.47 |
| bodi | 293 | 4 | 5 | 0.053 | 0.030 | 2.96 | 2.58 |
| flow | 730 | 3 | 6 | 0.020 | 0.013 | 1.18 | 1.12 |
| can | 289 | 4 | 0.029 | 2.45 | |||
| three | 140 | 2 | 0.034 | 2.76 | |||
| dimension (twice in query) | 237 | 2 | 0.034 | 4.26 | |||
| problem (twice in query) | 323 | 6 | 0.038 | 5.05 | |||
| reduc | 130 | 3 | 0.040 | 3.35 | |||
| Total | 0.347 | 0.260 | 20.28 | 25.03 |
Document 106 is a 44-token abstract that matches the query's most specific words, transverse, potential, and revolution, several times each. Document 1108 is a 170-token abstract that matches eight of the query's terms, mostly general ones such as can, three, problem, and reduce. In TF-IDF with cosine normalization, document 1108's long vector (length 11.5, against 6.2 for document 106) dilutes each of its matches. In BM25 each matching term adds its own saturated contribution, and the document's length only slows each term's saturation instead of dividing the whole score, so eight moderate matches beat five strong ones, and dimension and problem count twice because they appear twice in the query. Lengths and normalization are the whole difference: the two models agree on which words matter, and disagree about how to add them up. On query 173, about Lyapunov's method for linear differential equations with periodic coefficients, the opposite happens: TF-IDF ranks a short nonrelevant abstract first (average precision 0.583) while its rival ranks both relevant documents at the top (1.000).
How confident is the difference?
The $p$ values above come from a paired randomization test (see significance testing). For the closest pair, the 225 per-query differences in average precision are kept, and in each of 20,000 trials the sign of each difference is flipped at random, which is what the differences would look like if the labels "BM25" and "TF-IDF" were arbitrary. The $p$ value is the fraction of trials whose mean difference is at least as large in magnitude as the observed 0.0124: 0.075. Other ways of asking the same question agree:
- A paired $t$ test gives $t = 1.80$ with 224 degrees of freedom, $p = 0.073$.
- A bootstrap 95 percent confidence interval for the mean difference, from 10,000 resamplings of the queries, is $[-0.001, 0.026]$. It barely includes zero.
- Three comparisons were made, so the chance of at least one false positive is higher than 0.05. Holm's adjustment multiplies the smallest $p$ value by 3, the next by 2, and the largest by 1. The two comparisons with the BIM remain far below 0.05 and the closest pair stays at 0.075, so no conclusion changes.
"Not significant" here means that 225 queries are not enough to distinguish a difference of this size from noise in per-query results. It does not mean the models are equivalent. The interval says the true difference on queries like these is probably between a slight loss and a gain of about 0.026.
Tuning BM25
BM25 has two parameters. To tune them fairly, the 225 queries were split at random into a development set of 112 and a test set of 113, as test collections recommends. A grid search over $k_1 \in \{0.3, 0.6, 0.9, 1.2, 1.5, 2.0, 3.0\}$ and $b \in \{0, 0.25, 0.5, 0.75, 1\}$ on the development queries chose $k_1 = 1.5$ and $b = 0.75$. (The same grid, extended and evaluated on all 225 queries, is plotted in implementing BM25, where it shows a broad plateau around the defaults.)
| Development queries | Test queries | |
|---|---|---|
| Default ($k_1 = 1.2$, $b = 0.75$) | 0.316 | 0.297 |
| Tuned ($k_1 = 1.5$, $b = 0.75$) | 0.320 | 0.300 |
Tuning gained 0.004 on the queries it was tuned on and 0.003 on the unseen ones. The defaults are close to optimal for this collection, which is common: values near $k_1 = 1.2$ and $b = 0.75$ work well across many collections (Robertson and Zaragoza, 2009), which is why they became the defaults. The gain on development queries slightly exceeds the gain on test queries, the usual sign of a little overfitting.
Adding relevance information
The BIM was designed to use relevance judgments, and the comparison above denies it any. Relevance feedback supplies them. Using the judgments of each query's top 10 documents raised the BIM's effectiveness on the remaining documents by nearly half, and Rocchio feedback more than doubled that of TF-IDF. Pseudo-relevance feedback, which needs no judgments, raised BM25's MAP from 0.307 to 0.324 with settings fixed in advance, while harming 91 of the 225 queries.
Limits of the comparison
The results should be read with the limits of the experiment in mind:
- One small, old collection. Cranfield is 1,400 abstracts in one technical field, and its queries are unusually long and well formed. Results on web pages, news, or short keyword queries can differ.
- One text processing pipeline. Different stop lists, stemmers, or tokenizers would change every number, though usually not the ordering of models.
- Default settings. Each model was run largely with standard settings. A model tuned more carefully may close or widen a gap.
- No modern methods. Learned rankers and neural models, which exceed all four on collections they are trained for, are outside the course and this comparison.
Within those limits, the experiment shows each step of the course's progression doing what it was designed to do. Ranking answers the questions that strict Boolean matching cannot, term frequency and length normalization improve on binary weighting, and BM25 with default settings matches or slightly exceeds standard TF-IDF weighting without needing relevance information.
References
- Cleverdon, C. W. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19(6), 173–194.
- MacAvaney, S., Yates, A., Feldman, S., Downey, D., Cohan, A., and Goharian, N. (2021). Simplified data wrangling with ir_datasets. In Proceedings of SIGIR 2021, 2429–2436.
- Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, chapters 6, 8, and 11. Cambridge University Press.
- Robertson, S. E., and Walker, S. (1994). Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In Proceedings of SIGIR 1994, 232–241.
- Robertson, S., and Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389.