What Information Retrieval Is

A graduate student begins a thesis on the impact of climate change on Arctic biodiversity. The relevant literature is spread across ecology journals, climate assessments, government reports, and field datasets, and amounts to far more than one person can read. The student does not want every document that contains the words climate, Arctic, and biodiversity. A report that discusses Arctic weather in its first section and tropical biodiversity in its last contains all three words and is useless. The student wants documents about a relationship: how a changing climate is altering which species live in the Arctic and in what numbers. Some of the most useful documents may never use the word biodiversity at all, and speak instead of species ranges, population decline, or sea-ice habitat.

That gap, between the words a document contains and whether it serves the person asking, is the subject of information retrieval. This article defines the field, introduces its basic vocabulary, explains why the problem is hard, and distinguishes retrieval from the neighboring fields it is most often confused with.

A working definition

The standard textbook definition is that of Manning, Raghavan, and Schütze (2008, §1):

Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers).

Each qualifier matters. Material is usually documents, but the same methods apply to passages, products, images, people, and source code. Unstructured means the data has no schema that tells a program where the answer is: a news article has no field called main_claim. Satisfies an information need means success is defined by a person, not by a formal condition. Large collections means the system cannot simply read everything at query time; it must organize the collection in advance so that a request can be answered in a fraction of a second.

An older definition, Gerard Salton's from 1968, describes the field by its activities instead of its goal: it is "concerned with the structure, analysis, organization, storage, searching, and retrieval of information" (quoted in Croft, Metzler, and Strohman, 2010, ch. 1). The two definitions are complementary. Salton's lists the work a system does, and the textbook's names the standard that work is judged by. A useful summary combines them: information retrieval is the representation, organization, and search of large collections of human-language data so that people can find what they need in them.

The basic vocabulary

Five terms appear in nearly every discussion of retrieval.

An information need is the gap in someone's knowledge that prompts a search. It is internal and often only partly formed. A student preparing for a debate may need "the strongest arguments for and against nuclear power, with evidence for each." A patient may need to understand what a set of symptoms could mean. The need is what the system is ultimately trying to serve, and it is never directly observable.

A query is what the person gives the system to stand for the need. It is almost always a compressed and imperfect version of it. The debate student might type nuclear power pros and cons, is nuclear energy safe, or cost of nuclear plants per megawatt, and each query captures only part of the need. Web queries in particular are short, typically two or three words, so most of the need is left for the system to infer.

A document is the unit the system returns. It may be a web page, a scientific article, an email, a product listing, a paragraph, or a video. Choosing the unit is a design decision with real consequences. If an entire book is one document, a query matches it when its terms appear hundreds of pages apart. If every sentence is a document, no single unit contains enough context to be judged. Later articles return to this choice, which affects both indexing and evaluation.

A collection, or corpus, is the set of documents the system can search. Collections range from one person's files to a company's internal documents to the indexed web, which contains tens of billions of pages and changes continuously.

Relevance is the relationship between a document and an information need: whether the document helps. It is the central concept of the field and the hardest to pin down. A document can be on topic but already known to the reader, out of date, too technical, or from an unreliable source. Relevance also depends on who is asking. For the query jaguar, a page about the animal is relevant to a zoologist and irrelevant to someone shopping for a car. The article on relevance treats the concept in detail. For now, the key point is that relevance is defined with respect to the need, not the query. A document can match every query term and fail the need, or match none of them and serve it well.

From these five terms comes the basic shape of a retrieval system. It processes the collection in advance into an index, a data structure that maps terms to the documents containing them. When a query arrives, the system analyzes it the same way it analyzed the documents, uses the index to find candidate documents, scores each one by an estimate of its relevance, and returns a ranked list. The architecture article follows a small collection through each of these steps.

The figure makes one asymmetry plain. Everything the system works with (the query, the index, the scores) is on the right of the diagram, and the standard it is judged by is on the left, where the system cannot reach. The rest of the field follows from that gap. Retrieval models are methods for guessing well across it, and evaluation is the method for measuring, after the fact, how good the guesses were.

Why retrieval is hard

Ambiguity

Natural language does not map words to meanings one to one. Polysemy is one word with several meanings: python may refer to a snake or a programming language, and bank to a river's edge or a financial institution. Synonymy is several words with one meaning: car and automobile, heart attack and myocardial infarction. Polysemy produces false matches, documents that contain the query word in the wrong sense. Synonymy produces misses, relevant documents that express the idea in other words.

Queries add a further layer: underspecification. A two-word query rarely says which sense is intended, what kind of document is wanted, or how much the person already knows. java could be a request for a programming tutorial, a travel guide to the Indonesian island, or the history of coffee. The system must commit to a ranking anyway, which is why web engines often diversify the top results to cover several plausible readings.

Vocabulary mismatch

The most persistent difficulty in retrieval follows from synonymy at scale. In a series of experiments, Furnas, Landauer, Gomez, and Dumais (1987) asked people to name common objects and operations, such as the commands a text editor should have or the keywords for recipes. Across five such domains, they found that two people spontaneously chose the same main word for the same thing less than 20 percent of the time. An author describing "novel oncological therapies" and a searcher asking for "effective cancer treatments" may mean the same thing and share no words.

A system that matches only literal terms will therefore miss a large share of the relevant documents for almost any query. Much of the field's history consists of methods for closing that gap:

• normalizing words so that variants match, as stemming conflates retrieve, retrieval, and retrieving (covered in stemming and lemmatization);
• weighting terms so that the distinctive ones count most (TF-IDF and its successors);
• expanding queries with related terms, from thesauri or from the top results themselves (relevance feedback and query expansion);
• controlled vocabularies, in which each document is assigned standard subject headings, such as the U.S. National Library of Medicine's Medical Subject Headings;
• representations learned from data, from latent semantic indexing (see topic modeling) to the dense neural embeddings used in current systems.

None of these solves the problem completely. Each has costs, often some precision traded for recall or some efficiency for effectiveness, and whether a method helps depends on the collection and its users.

The opening example in miniature

The thesis student from the start of this article can be reduced to a collection small enough to inspect. Six one-sentence documents stand in for the literature. Three of them (B, C and D) serve the student's need, which is evidence about how a changing climate is altering Arctic species; the other three do not. The query is the obvious one, climate arctic biodiversity.

A system that demands every query word, as a Boolean AND does, retrieves A and B. Half of what it returns is useless, because A contains the three words without the relationship between them. And it finds only one of the three documents the student needs. In the vocabulary of precision and recall, the result has precision $1/2 = 0.50$ and recall $1/3 \approx 0.33$.

Ranking by the number of query words present does better on recall, since nothing is excluded, but the order is poor: A and B (three words each), then F (two), then C and E (one each), then D (none). The student must read all six documents to find all three useful ones. Each failure has a different cause. A is a false match: the words are present but not the relationship. C is a partial mismatch: the author wrote warming and range where the query said climate and biodiversity. D is a complete mismatch: it is entirely about Arctic species losing habitat to a warming climate and shares no word with the query at all. No weighting of the query's words can lift D above anything, because there is nothing to weight. Only a method that brings in words the searcher did not type, or that represents meaning rather than words, can find it.

The example is contrived, but the pattern is general, and the later topics take it apart one piece at a time. Term weighting makes the rarer and more telling words count for more. Relevance feedback would let the student mark B as good and add its words species and risk to the query; in a real collection many relevant documents share such words, though in this tiny one neither C nor D does. Representations learned from data can place polar bears losing sea-ice habitat near Arctic biodiversity without a shared word.

Effectiveness and efficiency at once

A retrieval system has two objectives that pull against each other. Effectiveness is the quality of the ranking: how well it places relevant documents above nonrelevant ones. Efficiency is the cost of producing it: response time, memory, disk, and machines. A web search engine is expected to answer within a few hundred milliseconds over billions of documents, so a ranking method that is more accurate but ten times slower may be unusable. Much of the engineering in later topics, such as compressed indexes, early termination, and multi-stage ranking, exists to make effective models affordable.

A rough calculation shows why the index is not optional. Suppose, for illustration, a collection of 50 billion web pages averaging 20 kilobytes of text each. That is $5 \times 10^{10} \times 2 \times 10^{4} = 10^{15}$ bytes, a petabyte. Reading it once at a sustained gigabyte per second takes $10^{6}$ seconds, about 11.6 days, for a single query. An inverted index answers the same query by reading only the lists of documents for the query's few terms, and at web scale even those lists are divided among thousands of machines. The sizes here are round numbers chosen to make the arithmetic easy, not measurements of any engine, but no plausible choice changes the conclusion: scanning is out of the question by a factor of millions.

What retrieval is not

Retrieval overlaps with several neighboring fields. The boundaries are not sharp, but they are worth drawing, because they determine which tool fits a given problem.

Databases

The clearest contrast is with database management systems. C. J. van Rijsbergen (1979, ch. 1) drew it as a distinction between data retrieval and information retrieval. The table below adapts his comparison:

A database query such as

SELECT name FROM employees
WHERE hire_date >= '2025-07-01' AND salary > 50000;

has a correct answer, and any system that returns a different set of rows is broken. A retrieval query such as employees who might lead the new analytics team has no single correct answer that follows from the query itself, only better and worse rankings. The difference shapes the data structures each field uses: databases rely on B-trees and hash indexes over typed fields, while retrieval systems rely on the inverted index over terms. It also shapes how each is evaluated. A database is tested for correctness. A retrieval system is measured statistically against human judgments, as described in evaluation.

The same collection often supports both kinds of request. The difference lies in whether the request can be checked mechanically against the stored data:

Each request on the left has an answer that any correct system returns exactly. Each request on the right needs a judgment about the meaning of text, and two careful people could disagree about the best answer.

In practice the two converge. Most relational databases now offer full-text search, and most search engines support structured filters, such as restricting results to a price range or a date. An online store's search box usually runs a retrieval model over product descriptions and database-style filters over attributes in the same request. The distinction remains useful as a way of identifying which part of a problem is exact and which part depends on judgment.

Natural language processing

Natural language processing (NLP) aims to analyze, understand, and generate human language: parsing, translation, summarization, sentiment analysis. Retrieval aims to find documents that serve a need, which may or may not require understanding them deeply. For decades the two fields developed largely apart. Retrieval research found that statistical methods, which treat a document as a weighted collection of words, outperformed the linguistically sophisticated methods available at the time for ranking documents. Early experiments in which syntactic phrases or semantic analysis were added to indexing often produced small or inconsistent gains.

That relationship has changed. Retrieval now uses NLP for query understanding, entity recognition, and spelling correction, and neural language models trained on large corpora produce document representations that capture similarity of meaning beyond surface words, though they can still miss exact terms that lexical matching would catch. In the other direction, retrieval supplies NLP systems with evidence: question answering and retrieval-augmented generation both begin by retrieving passages for a language model to read. The fields now share methods, but the questions they ask remain different. Retrieval asks which documents to show and in what order.

Recommender systems

A recommender system suggests items without an explicit query, using a person's past behavior, the behavior of similar people, or the features of the items themselves. A list of films "you might like" is a recommendation; a search for science fiction films about first contact is retrieval.

Recommendation is closely related to information filtering, in which a standing profile is matched against a stream of new documents using the same representations and matching functions as retrieval. Search applications describes filtering and its relationship to ad hoc search. Modern systems blur the line further. Search results are personalized using a person's history, and recommendations are retrieved by treating the person's profile as an implicit query.

Summary of the distinctions

The choice of tool follows from the problem. Precise questions over structured data belong to a database. Questions about the language itself, such as what a sentence means or how to translate it, belong to NLP. Proactive suggestions from patterns of behavior belong to a recommender. Retrieval is the right framing when the data is mostly text, the need can only be stated approximately, and the result must be ranked by how likely each item is to help. Many real systems combine all four.

What is at stake

Retrieval systems decide what people see. A search engine that ranks one source above another shapes what millions of people read about a medical treatment, a political candidate, or a historical event. Because few users look past the first page of results, the ranking function operates as a gatekeeper even when no person intends it to.

Several concerns follow from that role, and they appear throughout the later topics.

Bias. Rankings learned from data reproduce the patterns in the data. Safiya Noble (2018) documented searches for terms describing Black women that returned demeaning and sexualized results, an outcome produced by the interaction of popularity signals, advertising, and the content that existed to be ranked. Image searches for occupations have shown similar skews in gender and race.

Personalization and filter bubbles. Tailoring results to each person's history improves relevance by most measures, but Eli Pariser (2011) argued that it can also narrow the range of views a person encounters, since the system learns to show more of whatever was clicked before. How strongly this happens in practice is still disputed.

Fairness to providers. Rankings allocate attention, and attention has economic value. A system that always favors already-popular items concentrates exposure, so that newer or smaller sources struggle to be seen at all, regardless of their quality.

Opacity. A Boolean query or a TF-IDF score can be explained term by term. A ranking produced by a learned model with thousands of features, or by a neural network, cannot be decomposed so directly. Explanation methods such as feature attribution approximate why a model scored a document as it did, but they do not fully account for its behavior, which makes errors and biases harder to find and correct.

Olteanu and colleagues (2019) organized these concerns as FACTS-IR: fairness, accountability, confidentiality, transparency, and safety. The practical consequence for system builders is that optimizing a single effectiveness metric is not the whole job. Every design choice in a ranking system, including which signals to use, what to optimize, and how to evaluate, encodes a judgment about whose needs count.

The rest of this topic

The next article examines relevance closely, since every model and measure in the course approximates it. The history shows how the field arrived at its current methods, problem by problem. The architecture article then makes the vocabulary above concrete by following a small collection through a complete search engine.

References

• Croft, W. B., Metzler, D., and Strohman, T. (2010). Search Engines: Information Retrieval in Practice, chapter 1. Addison-Wesley.
• Furnas, G. W., Landauer, T. K., Gomez, L. M., and Dumais, S. T. (1987). The vocabulary problem in human-system communication. Communications of the ACM, 30(11), 964–971.
• Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, chapter 1. Cambridge University Press.
• Noble, S. U. (2018). Algorithms of Oppression: How Search Engines Reinforce Racism. New York University Press.
• Olteanu, A., Garcia-Gathright, J., de Rijke, M., Ekstrand, M. D., et al. (2019). FACTS-IR: Fairness, accountability, confidentiality, transparency, and safety in information retrieval. SIGIR Forum, 53(2), 20–43.
• Pariser, E. (2011). The Filter Bubble: What the Internet Is Hiding from You. Penguin Press.
• van Rijsbergen, C. J. (1979). Information Retrieval, 2nd edition, chapter 1. Butterworths.