Every technique in this course claims to improve retrieval, from stemming to BM25. Claims like these are settled by measurement, and information retrieval is unusual among the branches of computer science in how central measurement has been to its progress. Since the Cranfield experiments, the field has advanced mainly by building test collections, running systems against them, and comparing the results. In research, a method that cannot show an improvement on a test collection rarely gets adopted. This topic is about how that measurement works and where it can mislead.
The standard method has three parts: a collection of documents, a set of information needs, and judgments of which documents are relevant to each need. From these, a system's output can be scored. For a system that returns an unordered set, the scores are precision, the fraction of what was retrieved that is relevant, and recall, the fraction of what is relevant that was retrieved. For a ranked list, the scores reward placing relevant documents near the top: average precision, precision at a cutoff, and, when relevance comes in grades, nDCG. Each measure answers a slightly different question, and choosing one is a statement about what users care about.
The method rests on the simplified notion of relevance discussed in relevance. Judgments are made by assessors who disagree with one another, collections are too large for every document to be judged, and differences between systems can be smaller than the variation between queries. The later articles in this topic cover these problems. They show how agreement is measured, how pooling makes judging feasible, how statistical tests decide whether a difference is real, and how evaluation with live users complements test collections.
Articles
The method
- Test collections and the Cranfield paradigm: documents, topics, and judgments; how the Cranfield collection was built and what it contains, measured; the major collections from Cranfield to TREC and MS MARCO; and the rules for using them fairly, with an experiment on tuning against the test topics.
Measures
- Precision, recall, and the F measure: the contingency table, why accuracy is the wrong measure for retrieval, and why the F measure is a harmonic mean.
- Evaluating ranked results: precision-recall curves, interpolation, precision at k, R-precision, and reciprocal rank.
- Mean average precision: average precision for one query, its mean over many, and what it rewards.
- Discounted cumulative gain and nDCG: graded relevance, position discounts, normalization by the ideal ranking, and the two common gain formulas.
Judgments and comparisons
- Relevance judgments, pooling, and agreement: how judgments are collected, pooling simulated on Cranfield and its biases, the kappa statistic derived and worked, what substituting one assessor for another does to system rankings, and crowdsourced and model-generated judgments.
- Comparing systems with significance tests: per-query variation; the paired t-test, sign test, Wilcoxon test, randomization test, and bootstrap worked step by step; a real comparison on Cranfield; how many topics are needed; and multiple comparisons.
- Online evaluation: A/B tests and how many users they need, click-based measures and position bias, team-draft interleaving with an interactive demo, and how online and offline evaluation relate.
Doing it
- Implementing evaluation measures: the TREC formats for judgments and runs, a tested Python implementation of the main measures, and standard tools.
Related topics
- Relevance explains what relevance judgments stand for and why assessors disagree.
- A history of information retrieval describes the Cranfield experiments and TREC.
- The vector space model and probabilistic retrieval provide the ranking methods these measures compare.
Course materials
The course had no programming assignment for this topic. Evaluation was practiced by hand, computing measures for small rankings. The implementation article provides the code that an assignment would have produced.
References
- Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, chapter 8. Cambridge University Press.
- Sanderson, M. (2010). Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval, 4(4), 247–375.
- Voorhees, E. M., and Harman, D. K. (Eds.) (2005). TREC: Experiment and Evaluation in Information Retrieval. MIT Press.