Information retrieval is the study of finding, in a large collection, the items that answer a need for information. The course I took as CSC 790 covered its classical core in ten units: Boolean and ranked retrieval, indexing, the vector space and probabilistic models, evaluation, classification, clustering, web crawling and topic models. This collection continues from where the course stopped. Its direction is set by one purpose: storing and retrieving information for language models, which need text represented so that meaning can be matched, a way to search those representations quickly, a way to rank a short list precisely, and a way to hand what was found to the model and check what the model did with it.
The units are written as reference material. Each result that can be derived is derived on the page, each claim about how well a method works is cited to the paper that measured it, and the page says which is which. Worked examples are computed in full by scripts kept with the site's source (tools/experiments/). Many of them use this site's own search engine as the test bed, described in How This Search Works. Material that is still changing quickly is confined to named, published methods and carries the date it was written.
Units and articles that are not yet written appear below as plain text.
The course
The ten units of the course are listed on the course page and are the prerequisite for everything below.
- Foundations of Information Retrieval
- Boolean Retrieval
- Text Processing and Indexing
- The Vector Space Model
- Retrieval Evaluation
- Probabilistic Retrieval
- Text Classification
- Text Clustering
- Web Crawling
- Topic Modeling
Retrieval with learned representations
The route from sparse term vectors to learned ones, and from a single ranked list to a pipeline.
- Text Embeddings and Dense Retrieval: how text becomes a vector such that near vectors mean related text, how an encoder is trained for retrieval, and where one vector per passage fails.
- Vector Search: finding the nearest vectors among millions without comparing against all of them, with the accuracy given up stated as a number.
- Neural Ranking and Hybrid Retrieval: ranking a short list with a model that reads the query and passage together, the designs between that and a single vector, and combining lexical and learned retrieval.
- Retrieval-Augmented Generation: what happens after retrieval when the reader is a language model.
Engines and ranking
Classical subjects that the course introduced and that these units take to full depth.
- Efficient Query Processing: answering a query without scoring every posting, storing the index in less memory, and updating it without rebuilding.
- Learning to Rank Methods: the methods behind Learning to Rank, each derived in full, and learning from clicks.
Later
Two units are planned for later: knowledge stores and memory for language models, and queries and conversations.