Test Collections and the Cranfield Paradigm

Suppose a new stemmer is proposed, and the question is whether it improves retrieval. Trying it on a few queries and looking at the results is not enough: a few queries prove little, and the next person to try it will choose different ones. A fair comparison needs the same documents, the same information needs, and the same judgments of which documents satisfy each need, fixed in advance and shared by everyone. That combination is a test collection. Comparing systems by running them against test collections is called the Cranfield paradigm, after the experiments in which Cyril Cleverdon introduced it, run from the late 1950s to the mid-1960s (Cleverdon, 1967; see the history).

This article describes what a test collection contains, surveys the collections that shaped the field, and sets out the rules for using one fairly. The articles that follow define the measures computed from it.

The three components

A test collection has three parts (Manning, Raghavan, and Schütze, 2008, §8.1):

1. A document collection. A fixed set of documents, from a few thousand abstracts to hundreds of millions of web pages.
2. A set of information needs. Each need is written out in full, not just as a query. Fifty is a common minimum, for reasons explained in significance testing.
3. Relevance judgments. For each need, a judgment of whether each document, or each judged document, is relevant. The judgments are usually binary, relevant or not, and sometimes graded on a small scale. In TREC terminology the judgment file is the qrels, for query relevance judgments.

Needs, not queries

Relevance is judged against the information need, not the query. Manning, Raghavan, and Schütze give the example of the need

Information on whether drinking red wine is more effective at reducing your risk of heart attacks than white wine.

expressed as the Boolean query wine AND red AND white AND heart AND attack AND effective. A document that contains all six words but discusses, say, the effect of heart attacks on wine sales, is not relevant, because it does not address the need. A document that compares the effects of red and white wine on cardiovascular risk without using the word effective is relevant. Judging against the need is what makes a test collection measure whether a system finds useful documents, rather than whether it matches words.

For this reason TREC writes each information need as a structured topic with three fields:

• a title: a few words, like a typical query;
• a description: one sentence stating the need;
• a narrative: a paragraph saying exactly what counts as relevant and what does not.

Systems may be run with only the title, to imitate short user queries, or with the description too. The assessors always judge against the narrative.

Inside the Cranfield collection

The three components are easiest to understand in a real collection small enough to inspect completely. Cranfield is that collection, and the articles on comparing ranking models and relevance feedback run their experiments on it.

How it was built

Cleverdon, Mills, and Keen (1966, chapter 3) describe the construction in three stages. The method answered a specific problem: no existing set of real search questions came with people who could say which documents answered them.

1. Questions and first judgments from authors. The project collected 271 recent papers in high-speed aerodynamics and aircraft structures, called base documents, and wrote to each author. The author was asked to state the problem that had prompted the research as a search question, to add up to three supplementary questions, and to grade the relevance of the papers the base document cited, on a five-point scale. 182 authors replied with 641 questions. The cited papers, the base documents, and 209 further papers from similar sources made up the collection of 1,400 documents (pp. 19–21).
2. Screening every document against every question. The authors knew only the papers they had cited, but other documents in the collection could also be relevant. Five postgraduate students with a knowledge of aerodynamics examined every document against each of 361 selected questions, over 1,500 hours of work and in principle more than half a million individual decisions, and noted every document that was possibly relevant. They were told to be liberal, since a false alarm would be caught in the next stage (p. 22).
3. Final judgments from authors. The documents the students found, plus documents suggested by bibliographic coupling, were sent back to the question's author, who was the final arbiter of relevance (pp. 22–23).

The grading scale (Cleverdon, Mills, and Keen, 1966, p. 21) runs from 1, "references which are a complete answer to the question", through 2, "a high degree of relevance", 3, "useful, either as general background to the work or as suggesting methods", and 4, "minimum interest", to 5, "of no interest". Lower numbers are better, the reverse of most modern graded scales, which matters when grades are used as gains in nDCG.

Of the relevant documents the authors accepted, 1,250 came from their own citations, 592 from the students' screening, and 119 from bibliographic coupling only (Table 3.4). Two lessons follow. The authors' own knowledge covered less than two thirds of what they themselves judged relevant, so judging only the documents a single source proposes misses much of the relevant set. And even the exhaustive screening was not complete. Of the 711 additional relevant documents found by the students or by coupling, after a correction for documents accepted against a second question, the students had found 517, or 73 percent (p. 31). "Every document was judged" means every document was considered, not that every relevant document was found. The same problem returns, in a larger form, in the pooling used for large collections.

What the distributed collection contains

The version distributed for research today, loaded here with the ir_datasets library, has 1,400 documents and 225 of the questions. (Manning, Raghavan, and Schütze count 1,398 abstracts; two of the 1,400 records are essentially empty.) The script tools/experiments/cranfield/collection_stats.py measures it:

Two features matter for evaluation. First, most queries have few relevant documents, so a single relevant document moves a query's score a long way. With three relevant documents, finding one more raises recall by a third. Per-query scores are therefore coarse and vary widely between queries, which is why significance tests pair the scores by query. Second, the judgments are graded but skewed toward the middle grades, so a binary measure that counts grades 1 to 4 as relevant treats a "minimum interest" reference the same as a complete answer.

Each query also has exactly one entry with the code −1. The notes distributed with the judgments do not define it: they quote Cleverdon's five grades and say only that no grade-5 judgments are included. The documentation of the ir_datasets library (MacAvaney et al., 2021) labels −1 "references of no interest", Cleverdon's wording for grade 5, but the same table also reverses grades 1 to 4 relative to the distributed notes, so it is not an independent account of what the code meant. For query 1 the −1 document is 486, similarity laws for aerothermoelastic testing, whose title restates the query. A plausible reading, which no source we have found states, is that −1 marks the base document from which the question was drawn, which Cleverdon's team deleted from the results when testing that question (1966, p. 21). The data fit that reading. The 225 entries name only 128 distinct documents, as expected when one author's paper gave rise to several questions, and BM25 ranks the −1 document first for 91 of the 225 queries and in the top ten for 162, which is what the paper a question was written from would be expected to do. The experiments on this site treat it as nonrelevant, as the distributed judgments do, so systems lose some credit at rank 1 that the original experiments would not have charged.

Query 1 shows what the judgments look like:

what similarity laws must be obeyed when constructing aeroelastic models of heated high speed aircraft.

Twenty-eight of the 1,400 documents are relevant to it, and every other document was considered and found not to be. The narrative that a TREC topic would state is missing; what counted as relevant was decided by the author who asked the question, as the need behind the question rather than its words.

Major collections

Sizes are from Manning, Raghavan, and Schütze (2008, §8.2), except MS MARCO (Bajaj et al., 2016).

The collections illustrate a trade-off between depth and breadth of judgment. Cranfield judged every document for every query, which is possible only for a small collection. TREC judged a pool of the top-ranked documents from many systems for each of about fifty topics a year (Voorhees and Harman, 2005), described in relevance judgments, pooling, and agreement. MS MARCO judged very few documents for each of a very large number of queries, enough to train learned rankers, which need many queries. Shallow judgments make it unsafe to treat unjudged documents as nonrelevant, which shapes the measures used with such collections.

Using a collection fairly

A test collection is a shared instrument, and it can be misused in ways that inflate results without anyone intending it.

Separate development from testing. Every retrieval method has parameters: BM25's $k_1$ and $b$, the slope of pivoted normalization, the number of feedback terms. If they are tuned on the same topics on which results are reported, the reported score measures how well the method fits those topics, not how well it will perform on new ones. The standard practice is to tune on a development set of topics and report results on a separate test set (Manning, Raghavan, and Schütze, 2008, §8.1), ideally used once.

How large is the effect? An experiment on Cranfield (collection_stats.py) makes it concrete. BM25 has two parameters, and a grid of 7 values of $k_1$ (0.3 to 3.0) and 5 values of $b$ (0 to 1) gives 35 settings. On all 225 queries the best setting, $k_1 = 3.0$ and $b = 0.5$, reaches a MAP of 0.312, against 0.307 for the usual default $k_1 = 1.2$, $b = 0.75$. Reporting that 0.005 gain would be reporting a number chosen because it was the largest of 35. To see how much of it is real, the queries were split at random into a tuning half (112 queries) and a held-out half (113), the best of the 35 settings was chosen on the tuning half, and its gain over the default was measured on both halves. Over 30 random splits:

Most of the apparent gain was selection, not improvement. The setting that wins on the tuning half wins partly because it suits the particular queries in that half, and those queries are not repeated in the other half. The figure shows a pattern that makes the effect hard to notice from inside one experiment: the ten splits with the largest tuning gains averaged −0.0019 on their held-out halves, and the ten with the smallest averaged +0.0045. Part of this is mechanical, since the two halves always add up to the same 225 queries, so a setting that does unusually well on one half must do correspondingly less well on the other. That is the point: an unusually large gain on the tuning topics is a sign of luck in the split as much as of a good setting. A large improvement found by searching over many settings is evidence mostly of the search.

Beware repeated reuse. Even with that discipline, a research community that tests hundreds of ideas on the same collection will eventually find some that work there by chance. Results that hold across several collections are more trustworthy than a large gain on one.

Report per-query results, not only averages. Systems differ far more across queries than between each other, and an average can hide a method that helps a few queries greatly and harms many slightly. On Cranfield, Porter stemming raises BM25's MAP from 0.282 to 0.307, but it makes 83 of the 225 queries worse, one of them by 0.45 in average precision, while the scores of either system vary between queries with a standard deviation of about 0.23. Significance tests use the per-query results, and that article works through this comparison.

Check for unjudged documents. A collection judged by pooling has judgments only for the documents that its original systems retrieved. A new system may retrieve relevant documents that nobody judged, and standard measures count them as nonrelevant. Reporting how many of a new system's top documents are unjudged reveals whether this is happening. In the pooling simulation in relevance judgments, runs that contributed to a shallow pool had none of their top ten documents unjudged, while the other runs had between 13 and 38 percent unjudged, and the weakest run 88 percent.

What a test collection measures

A test collection measures how well a system ranks documents against fixed judgments of topical relevance, averaged over a set of needs. That is a simplified version of what users care about, as relevance explains. It leaves out how fast results arrive, how well snippets help, whether the user's need changed during the session, and whether the user was satisfied. Manning, Raghavan, and Schütze (2008, §8.6) call the true goal user utility, and relevance judgments a proxy for it. The proxy has served the field well. Methods that improve scores on test collections have generally improved real systems, though, as noted in relevance, user studies show the correspondence is imperfect. Complementary methods that measure behavior directly are covered in online evaluation.

References

• Bajaj, P., Campos, D., Craswell, N., et al. (2016). MS MARCO: A human generated machine reading comprehension dataset. arXiv:1611.09268 (version 3, 2018).
• Cleverdon, C. W. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19(6), 173–194.
• Cleverdon, C. W., Mills, J., and Keen, M. (1966). Factors Determining the Performance of Indexing Systems, Volume 1: Design, Part 1: Text. Aslib Cranfield Research Project, Cranfield.
• MacAvaney, S., Yates, A., Feldman, S., Downey, D., Cohan, A., and Goharian, N. (2021). Simplified data wrangling with ir_datasets. In Proceedings of SIGIR 2021, 2429–2436. Cranfield documentation: https://ir-datasets.com/cranfield.html
• Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, sections 8.1, 8.2, and 8.6. Cambridge University Press.
• Voorhees, E. M., and Harman, D. K. (Eds.) (2005). TREC: Experiment and Evaluation in Information Retrieval. MIT Press.