Information Retrieval

A collection that continues the information retrieval course on this site past its last unit, toward storing and retrieving information for language models, with embeddings and dense retrieval, vector search, neural ranking, retrieval-augmented generation, efficient query processing and learning to rank. Units not yet written are listed as plain text.

Information retrieval is the study of finding, in a large collection, the items that answer a need for information. The course I took as CSC 790 covered its classical core in ten units: Boolean and ranked retrieval, indexing, the vector space and probabilistic models, evaluation, classification, clustering, web crawling and topic models. This collection continues from where the course stopped. Its direction is set by one purpose: storing and retrieving information for language models, which need text represented so that meaning can be matched, a way to search those representations quickly, a way to rank a short list precisely, and a way to hand what was found to the model and check what the model did with it.

The units are written as reference material. Each result that can be derived is derived on the page, each claim about how well a method works is cited to the paper that measured it, and the page says which is which. Worked examples are computed in full by scripts kept with the site's source (tools/experiments/). Many of them use this site's own search engine as the test bed, described in How This Search Works. Material that is still changing quickly is confined to named, published methods and carries the date it was written.

Units and articles that are not yet written appear below as plain text.

The course

The ten units of the course are listed on the course page and are the prerequisite for everything below.

Retrieval with learned representations

The route from sparse term vectors to learned ones, and from a single ranked list to a pipeline.

  • Text Embeddings and Dense Retrieval: how text becomes a vector such that near vectors mean related text, how an encoder is trained for retrieval, and where one vector per passage fails.
  • Vector Search: finding the nearest vectors among millions without comparing against all of them, with the accuracy given up stated as a number.
  • Neural Ranking and Hybrid Retrieval: ranking a short list with a model that reads the query and passage together, the designs between that and a single vector, and combining lexical and learned retrieval.
  • Retrieval-Augmented Generation: what happens after retrieval when the reader is a language model.

Engines and ranking

Classical subjects that the course introduced and that these units take to full depth.

Later

Two units are planned for later: knowledge stores and memory for language models, and queries and conversations.