Information retrieval is the study of finding material, usually text, that satisfies an information need from within a large collection. The definition is simple, but each of its parts is difficult. The need exists in someone's head and reaches the system only as a few typed words. The material is written by thousands of authors who never anticipated the query. "Satisfies" is a human judgment that no system can observe directly. Retrieval systems manage this uncertainty instead of removing it: they rank documents by an estimate of how likely each one is to be useful, and they are judged by how well that ranking agrees with people.
This topic surveys the field before the rest of the course examines its parts. It defines the vocabulary used everywhere else (information need, query, document, collection, relevance) and separates retrieval from its neighbors: databases, natural language processing, and recommender systems. It traces the field's history as a series of problems and the ideas that answered them, many of which are still in use. It then follows a document and a query through a search engine, from acquisition and indexing to ranking and presentation, and surveys where search is used, from the open web to law, medicine, and a company's internal documents. Each later topic in the course takes one stage of that pipeline and treats it in depth.
Articles
The field
- What information retrieval is: information needs, queries, documents, and collections; how retrieval differs from database querying; ambiguity and vocabulary mismatch; and what is at stake when systems decide what people see.
- Relevance: the concept every model approximates and every evaluation measures, why it resists definition, and how people's information needs change as they search.
- A history of information retrieval: from microfilm selectors and punched cards to Cranfield, SMART, TREC, link analysis, and learned rankers, and why old ideas keep surviving inside new systems.
How a search engine works
- The architecture of a search engine: the indexing process and the query process, a small collection carried through the whole pipeline, and how the contents of the index limit what ranking can do.
- How a web search engine answers a query: navigational, informational, and transactional queries; candidate generation, scoring, duplicate removal, and snippets; and the ways engines rewrite queries.
- Search at scale: caching, replication, partitioning the index by document or by term, multi-stage ranking, tail latency, and the index as a periodically refreshed snapshot, with the sizes worked out for a billion-document index.
Where search is used
- Search applications: desktop and web search, routing and filtering, classification and clustering, summarization, information extraction, question answering, expert finding, and multimedia retrieval.
- Search in professional domains: legal, medical, patent, and scholarly search, where recall, authority, and controlled vocabularies matter more than speed.
- Enterprise, personal, and product search: access control, sparse link structure, known-item search, attribute queries, and ranking for purchase.
Building one
- Designing a modular search engine: separating text analysis, indexing, query processing, and ranking behind interfaces so that each can be replaced and tested on its own, with this site's own search engine as a second example.
What comes next
The rest of the course follows the pipeline. Text processing and indexing turns raw documents into terms and builds the inverted index. Boolean retrieval answers exact-match queries against it. The vector space model introduces graded scores and term weighting. Evaluation defines how to tell whether one ranking is better than another. Probabilistic retrieval derives ranking from the probability of relevance. Classification and clustering organize documents rather than rank them, web crawling gathers the collection in the first place, and topic modeling represents documents by latent themes instead of words.
Course materials
The course project built a search engine in stages, one component per topic. My versions of those components are in the information-retrieval repository on GitHub.
References
- Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, chapter 1. Cambridge University Press.
- Croft, W. B., Metzler, D., and Strohman, T. (2010). Search Engines: Information Retrieval in Practice, chapters 1–2. Addison-Wesley.
- van Rijsbergen, C. J. (1979). Information Retrieval, 2nd edition. Butterworths.
- Sanderson, M., and Croft, W. B. (2012). The history of information retrieval research. Proceedings of the IEEE, 100 (Special Centennial Issue), 1444–1451.