Relevance

Consider three documents returned for the query jaguar habitat loss. The first is a 2009 conservation report on deforestation in the jaguar's range. The second is a recent peer-reviewed study on the same subject, written for ecologists and dense with population models. The third is a short magazine piece that summarizes the study. All three are about the topic of the query, yet whether each one is relevant depends on who is asking. A conservation biologist who has already read the 2009 report gains nothing from it and wants the study. A high-school student writing a report cannot use the study and wants the magazine piece. A journalist checking a claim may need all three.

Every retrieval model in this course is a method for estimating relevance, and every evaluation measure compares a system's ranking to human judgments of it. Yet relevance is not a property of a document. It is a relationship between a document and a person's need at a particular moment, and it resists precise definition. This article describes what is known about that relationship, why the field nevertheless operates with a simplified version of it, and what the simplification costs.

From need to query

The previous article distinguished the information need from the query. Robert Taylor (1968), studying how librarians interview patrons at a reference desk, described four levels of a need, each a narrower expression of the one before:

1. The visceral need: an unexpressed and perhaps unconscious sense that something is missing.
2. The conscious need: a mental description of the gap, often vague and rambling.
3. The formalized need: a rational statement of the question.
4. The compromised need: the question as it is actually put to a librarian or a system, shaped by what the person believes the system can answer.

A query typed into a search box is a compromised need. Its form reflects the searcher's beliefs about how the system works as much as the need itself. People who have learned that search engines match keywords type keywords, and people who expect a conversational system type questions.

Nicholas Belkin (1980) pushed the point further. His anomalous state of knowledge (ASK) hypothesis holds that people often search because their knowledge is incomplete, so they cannot fully specify what they need. A known-item search, for a document the searcher can already name, is a different case; the ASK hypothesis concerns searches that begin from a gap the searcher cannot yet describe. Retrieval systems therefore cannot assume that the query fully describes the need. At best it is a partial description, and the system's job includes helping the person discover what they are looking for.

Needs change during a search

The classical model of retrieval assumes that a need is fixed: a person submits one query, the system returns one ranking, and the ranking is judged against the need as it was at the start. Observations of real searchers contradict this.

Marcia Bates (1989) described searching as berrypicking. A person starts with a broad query, reads a few results, learns new terms and ideas from them, follows a citation, reformulates, and gradually gathers useful pieces from many sources. The need evolves with each step, and the useful result of the search is the accumulated collection, not any single ranked list. Carol Kuhlthau (1991), studying students researching assignments, found that the process also has an emotional course: uncertainty and anxiety are highest early, when the topic is still unformed, and confidence rises as a focus emerges. A document that is useless during exploration may become important once the person knows what they are looking for, and the reverse.

These observations do not invalidate ranked retrieval, but they explain several features of practical systems. Query suggestions, related searches, and snippets support reformulation. Relevance feedback uses documents the searcher has judged to revise the query. Session-aware ranking treats a sequence of queries as evidence about a single developing need. They also mark a limit of laboratory evaluation, which typically scores one query at a time against fixed judgments.

Kinds of relevance

Because relevance depends on so much, researchers have distinguished several kinds of it. Tefko Saracevic's reviews of the literature (1975, 2007) and Stefano Mizzaro's history (1997) are the standard guides. The distinctions below are the ones most useful for understanding retrieval systems.

System relevance, also called algorithmic relevance, is relevance as a system estimates it, usually expressed as the score its ranking function assigns to a document for a query. It is computed, objective in the sense of being reproducible, and only a proxy for the others. Its scale depends on the model: a cosine similarity of 0.8 and a BM25 score of 8 are not comparable, and neither is a probability.

Topical relevance is whether a document is about the subject of the query. William Cooper (1971) gave an early formal account, which he called logical relevance, in terms of whether a document contains a statement that bears logically on the question. Logical relevance is stricter than topicality: a document can share the subject of a question without containing anything from which an answer follows. Topicality is the kind of relevance most classical retrieval models target, because word overlap is evidence of shared subject matter.

Situational relevance, a term from Patrick Wilson (1973), is whether a document bears on the person's actual situation: their task, their decision, and what they already know. A document can be topically relevant and situationally useless because the reader already knows its contents, or because it is too elementary or too advanced.

Cognitive relevance, also called psychological relevance (Harter, 1992), is whether the document changes the reader's state of knowledge. This definition makes relevance depend on reading order: the second of two documents saying the same thing is less relevant than the first, because the reader has already learned what it says.

Affective relevance refers to the effect a document has on the reader's motivation, interest, or satisfaction.

When people are asked how they actually judge documents, they name many criteria besides topic. In a study of academic users, Carol Barry (1994) found criteria including the depth and scope of a document, its novelty to the reader, the reputation of its author and source, its recency, its clarity, and whether it was available to obtain. Relevance as people experience it is a combination of these factors, and they weight them differently for different tasks.

The opening example, judged several ways

The three documents from the start of this article make the distinctions concrete. Suppose the conservation biologist has already read the 2009 report, the student is writing for a general audience, and the journalist is checking a figure quoted in the press.

Every cell in the Topical column is "yes": an assessor judging topicality alone would mark all three documents relevant, and so would a system that matches jaguar, habitat and loss. The other columns are situational judgments, and on every one of the three documents at least two of the readers disagree. No single ranking of the three is best for all three people, and no system that sees only the query can tell which person is asking.

Cognitive relevance adds an ordering effect. Once the biologist has read the study, the magazine summary carries nothing new, so its value depends on what came before it in the list. Had the magazine piece come first, it would have told the biologist something, and the study would still have added detail. Relevance in this sense belongs to a sequence of documents, not to a document alone, which is exactly what the field's standard simplification, described next, sets aside.

The simplification the field relies on

A system cannot observe a person's situation, prior knowledge, or emotional state. To make retrieval tractable to build and to measure, the field has worked mostly with a narrower notion, made operational by the Cranfield experiments of the 1960s and carried into every test collection since (see the history). In this version:

• relevance is topical, judged against a written statement of the need rather than the need itself;
• it is judged by an assessor, who is often not the person who had the need;
• it is binary (relevant or not) or graded on a small scale (for example, not relevant, relevant, highly relevant);
• each document is judged independently of the others.

Independence here means that an assessor judges each document without regard to the other documents seen. It is a different assumption from the statistical independence of terms used by the binary independence model.

The last assumption is the most consequential for ranking. If relevance is a property of each document alone, and each relevant document is worth the same whatever is ranked above it, the best ranking orders documents by their individual probability of relevance, which is the Probability Ranking Principle (Robertson, 1977). Cognitive relevance violates the assumption. Ten near-duplicate copies of the best document would each be judged relevant, but a ranking that lists all ten serves the reader worse than one that lists the best document followed by nine different ones. Methods for result diversification and novelty address this directly, and so do evaluation measures that penalize redundancy.

The simplification is not naive. It makes it possible to compare two systems on the same queries with the same judgments, which is the basis of the experimental method described in evaluation. Whether systems that score better under the simplified definition also serve real users better is less certain. User studies have found that improvements measured this way do not always produce better performance on real search tasks (Hersh et al., 2000; Turpin and Hersh, 2001), a gap the evaluation topic examines.

People disagree, and it matters less than expected

If relevance depends on the judge, two assessors given the same documents and the same need should disagree. They do. Ellen Voorhees (2000) had TREC topics judged by additional assessors and compared their judgments. The sets of documents that different assessors marked relevant overlapped by less than half: for pairs of assessors on TREC-4 topics, the size of the intersection divided by the size of the union averaged between about 0.42 and 0.49.

A small constructed example shows what an overlap of that size means. Two assessors judge the same twelve pooled documents, labelled a to l, for one topic. Assessor A marks seven relevant, assessor B six:

The overlap is $4/9 \approx 0.44$, inside the range Voorhees measured. The assessors agree on only four of the nine documents that either of them thought relevant, and each would call the other's judgments wrong on five documents. Now score three systems by precision at 5, the fraction of each system's top five that is relevant, under each assessor's judgments:

S2's score falls by a third when the judges change, yet the order of the systems is S1, S2, S3 under both. That is the pattern Voorhees found at scale, across many topics and real assessors. The example is built to show it and proves nothing by itself; relevance judgments, pooling, and agreement repeats the experiment on the Cranfield collection with simulated second assessors. There, random disagreement swaps only systems with nearly equal scores, but an assessor whose disagreements happen to favour one system can reorder the comparison.

The striking result was what this did to system comparisons. When Voorhees ranked the participating retrieval systems by their scores under each assessor's judgments, the ranking of systems barely changed. Absolute scores moved substantially, but the order of systems was stable, apparently because the disagreements did not systematically favor some systems over others. This is why test collections remain useful even though their judgments are not "true" in any absolute sense: they support comparison between systems, not absolute measurement of quality. In measurement terms, Voorhees's result concerns reliability, whether comparisons are stable when the judges change. The user studies of Hersh and Turpin concern validity, whether those comparisons predict how well people complete real tasks.

Relevance judgments, pooling, and agreement returns to this with the kappa statistic for measuring assessor agreement, and with pooling, the method that makes judging large collections feasible by judging only the documents that some system ranked highly.

What systems can observe

Since the need itself is hidden, a working system estimates relevance from signals it can observe. Each signal approximates some kind of relevance and misses others.

Click data deserves particular caution. People click the top results more often whatever they contain, so a raw click count largely measures where a result was shown. Joachims and colleagues (2005) showed with eye-tracking that clicks are informative mainly as relative judgments: because people tend to scan results from the top down, a click on the third result after passing over the first two suggests that the third was preferred to the two above it. Treating clicks as absolute relevance labels trains a system to repeat its own past rankings.

Joachims and colleagues turned that observation into explicit rules for reading clicks (2005, §5.3). The simplest, "Click > Skip Above", says that a clicked result is preferred to every result ranked above it that was not clicked. Their own example is a ranking of seven results in which the user clicks the first, third and fifth.

The rule extracts three preferences and no absolute judgments. It does not say that l1 is relevant, only that nothing suggests it was worse than what lay below it, and it says nothing at all about results below the last click. In the first phase of their study, preferences extracted this way agreed in direction with judges' explicit preferences between result summaries 80.8 percent of the time. Chance would give 50 percent, and two judges agreed with each other 89.5 percent of the time, so clicks read as relative preferences come fairly close to explicit judgments. Relative evidence of this kind is what online evaluation and click-trained rankers are built on.

Where relevance appears in the course

Each later topic handles relevance in a different way, and the differences are a useful map of the course.

• Boolean retrieval treats relevance as satisfaction of a logical condition. A document either meets the query or does not, which matches relevance only when the searcher can state the need exactly.
• The vector space model replaces the condition with a degree of similarity between query and document. Similarity is a heuristic stand-in for topical relevance, with no explicit model of what relevance is.
• Evaluation makes relevance operational through assessors' judgments and builds measures such as precision, recall, and nDCG on top of them. Graded judgments enter here, since nDCG rewards placing highly relevant documents above marginally relevant ones.
• Probabilistic retrieval models relevance directly, as a random variable whose probability the system estimates from term statistics. It is the only family of ranking models in the course that names relevance in its equations.
• Relevance feedback uses relevance judgments from the searcher, rather than an assessor, to revise the query. It is the clearest case of a system responding to the searcher's own sense of relevance during a search.
• Text classification reverses the setting: relevance to a standing category is learned from labeled examples, as in spam filtering or routing documents to the right team.

Ultimately a system is built to be useful, and relevance is the measurable stand-in for usefulness. Manning, Raghavan, and Schütze (2008, §8.6) call the true objective user utility, and note that relevance is a proxy for it. The rest of the course works mostly with topical relevance, binary or graded, because that is what can be modeled and measured. The kinds of relevance it leaves out mark where those models are most likely to fail.

References

• Barry, C. L. (1994). User-defined relevance criteria: An exploratory study. Journal of the American Society for Information Science, 45(3), 149–159.
• Bates, M. J. (1989). The design of browsing and berrypicking techniques for the online search interface. Online Review, 13(5), 407–424.
• Belkin, N. J. (1980). Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science, 5, 133–143.
• Cooper, W. S. (1971). A definition of relevance for information retrieval. Information Storage and Retrieval, 7(1), 19–37.
• Harter, S. P. (1992). Psychological relevance and information science. Journal of the American Society for Information Science, 43(9), 602–615.
• Hersh, W., Turpin, A., Price, S., Chan, B., Kraemer, D., Sacherek, L., and Olson, D. (2000). Do batch and user evaluations give the same results? In Proceedings of SIGIR 2000, 17–24.
• Joachims, T., Granka, L., Pan, B., Hembrooke, H., and Gay, G. (2005). Accurately interpreting clickthrough data as implicit feedback. In Proceedings of SIGIR 2005, 154–161.
• Kuhlthau, C. C. (1991). Inside the search process: Information seeking from the user's perspective. Journal of the American Society for Information Science, 42(5), 361–371.
• Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, sections 8.1, 8.5, and 8.6. Cambridge University Press.
• Mizzaro, S. (1997). Relevance: The whole history. Journal of the American Society for Information Science, 48(9), 810–832.
• Robertson, S. E. (1977). The probability ranking principle in IR. Journal of Documentation, 33(4), 294–304.
• Saracevic, T. (1975). Relevance: A review of and a framework for the thinking on the notion in information science. Journal of the American Society for Information Science, 26(6), 321–343.
• Saracevic, T. (2007). Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: Nature and manifestations of relevance. Journal of the American Society for Information Science and Technology, 58(13), 1915–1933.
• Taylor, R. S. (1968). Question-negotiation and information seeking in libraries. College and Research Libraries, 29(3), 178–194.
• Turpin, A. H., and Hersh, W. (2001). Why batch and user evaluations do not give the same results. In Proceedings of SIGIR 2001, 225–231.
• Voorhees, E. M. (2000). Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management, 36(5), 697–716.
• Wilson, P. (1973). Situational relevance. Information Storage and Retrieval, 9(8), 457–471.