The vector space model replaced the Boolean model as the default in research systems in the 1970s and became the foundation of ranked retrieval. Its central picture of documents and queries as directions in a space of terms, compared by the angle between them, is still how many people first think about search, and it returns, in a new form, in the dense embeddings of neural retrieval. It also rests on assumptions that are known to be false, and its failures are as instructive as its successes. This article assesses the model, describes the best-known way of improving its queries, and traces where its ideas went.
Strengths
- Ranking and partial matching. Every document receives a graded score, so results can be ordered and documents that match part of a query are not lost. This removes the feast-or-famine problem of Boolean retrieval.
- Term weighting. The model provides a framework in which the importance of terms, through TF-IDF and its variants, can be expressed and tested. Weighting turned out to matter more than almost any other design choice.
- Simplicity and speed. Scores are sums of products over shared terms, computed directly from the inverted index as described in computing vector scores.
- Symmetry. The same measure compares queries with documents and documents with each other, supporting similar-document search, clustering, and classification.
- Empirical strength. Well-tuned vector space systems, such as SMART with pivoted normalization, were among the strongest performers in early TREC evaluations.
The independence assumption
The model treats every term as a separate dimension, at right angles to every other. Geometrically, that says terms are unrelated: car is no closer to automobile than to banana. A query about cars has a cosine of zero with a document that only says automobile. This is the vocabulary mismatch problem in geometric form, and the model can do nothing about it without help.
The running example of this topic, introduced in TF-IDF weighting, has the problem built in. Its page $d_3$ is about auto insurance and says auto three times, but the query best car insurance asks for car, and the auto axis is at right angles to the car axis. Of $d_3$'s weight, only the parts on best and insurance count toward its cosine with the query, 0.516 under TF-IDF weighting (see cosine similarity). Its three occurrences of auto lengthen its vector and so, if anything, lower its score.
The model's own data contradicts the assumption. Read the count matrix by rows instead of columns, so that each term is a vector over the four documents: car is $(1, 5, 0, 3)$ and auto is $(0, 1, 3, 0)$. Their cosine is 0.267, not 0, because they occur together in the car review. In a large collection, words with related meanings co-occur far more than unrelated ones, and that co-occurrence is exactly what the orthogonal axes ignore.
The opposite problem, polysemy, is equally invisible. Jaguar the animal and Jaguar the car share a dimension, so a query about one matches documents about the other.
Raghavan and Wong (1986) made a related criticism: treating terms as orthogonal basis vectors is inconsistent with the evident fact that terms are correlated.
No theory of relevance
A more fundamental limitation concerns relevance itself. The vector space model says what to compute, a similarity, but not why similarity should predict relevance. As a result, there is no principled answer to questions such as how term frequency should be scaled, what IDF should be, or how long documents should be treated. Each was settled by trying alternatives on test collections, as the history of weighting variants shows. Probabilistic retrieval starts instead from the probability that a document is relevant, and it derives a weighting closely resembling TF-IDF, along with explanations for its components. The two approaches converge in practice. The difference is whether the formula comes from a model of relevance or from experiment.
Improving the query: Rocchio relevance feedback
If the query vector points in the wrong direction, perhaps because the searcher used different words from the relevant documents, one remedy is to move it. Relevance feedback asks the searcher to mark some retrieved documents as relevant or not, and uses those judgments to revise the query. In the vector space model, the classic method is due to J. J. Rocchio (1971), who developed it in the SMART system. The revised query is
$$\vec{q}_m = \alpha\, \vec{q}_0 + \beta\, \frac{1}{\lvert D_r \rvert} \sum_{\vec{d} \in D_r} \vec{d} \;-\; \gamma\, \frac{1}{\lvert D_{nr} \rvert} \sum_{\vec{d} \in D_{nr}} \vec{d},$$
where $\vec{q}_0$ is the original query, $D_r$ and $D_{nr}$ are the documents judged relevant and nonrelevant, and $\alpha$, $\beta$, $\gamma$ set how much each part counts. The query moves toward the centroid, the average vector, of the relevant documents and away from the centroid of the nonrelevant ones. Typical values are $\alpha = 1$, $\beta = 0.75$, and $\gamma = 0.15$, weighting positive evidence more than negative. Negative components of the result are set to zero (Manning, Raghavan, and Schütze, 2008, §9.1.1).
A worked example
A searcher interested in the animal types jaguar. Over the terms (jaguar, car, cat, speed), the query is $\vec{q}_0 = (1, 0, 0, 0)$. The searcher marks two results relevant, $(1, 0, 1, 1)$ and $(1, 0, 1, 0)$, both about the cat, and one nonrelevant, $(1, 1, 0, 1)$, about the car. With $\alpha = 1$, $\beta = 0.75$, and $\gamma = 0.15$:
| Term | $\alpha \vec{q}_0$ | $\beta \times$ relevant centroid | $\gamma \times$ nonrelevant centroid | $\vec{q}_m$ |
|---|---|---|---|---|
| jaguar | 1 | 0.75 × 1 = 0.75 | 0.15 × 1 = 0.15 | 1.60 |
| car | 0 | 0 | 0.15 × 1 = 0.15 | −0.15 → 0 |
| cat | 0 | 0.75 × 1 = 0.75 | 0 | 0.75 |
| speed | 0 | 0.75 × 0.5 = 0.375 | 0.15 × 1 = 0.15 | 0.225 |
The revised query keeps jaguar, adds cat with a substantial weight, gives speed a small one, and assigns car a negative weight that is clipped to zero. Documents about the animal that never use the word jaguar can now be retrieved through cat, which partly addresses vocabulary mismatch.
Rocchio on the running example
Return to the query best car insurance and the four documents of this topic's running example. Suppose the searcher sees the ranked list and marks the headline $d_1$ and the auto-insurance page $d_3$ relevant and the car review $d_2$ nonrelevant. With every vector weighted by TF-IDF and normalized to unit length (the normalized table in cosine similarity), and the same $\alpha$, $\beta$ and $\gamma$:
| Term | $\alpha \vec{q}_0$ | $\beta \times$ centroid of $d_1, d_3$ | $\gamma \times d_2$ | $\vec{q}_m$ |
|---|---|---|---|---|
| auto | 0 | 0.310 | 0.061 | 0.249 |
| best | 0.358 | 0.043 | 0.051 | 0.350 |
| car | 0.358 | 0.106 | 0.127 | 0.336 |
| insurance | 0.862 | 0.461 | 0 | 1.323 |
| loan | 0 | 0 | 0 | 0 |
| rate | 0 | 0.254 | 0 | 0.254 |
(Rounded to three decimals from unrounded values.) The revised query has learned auto from $d_3$, which is the vocabulary mismatch being repaired, and it has strengthened insurance, which both relevant documents share. It has also picked up rate, which appears in the relevant headline only by accident of its wording; feedback adds whatever the relevant documents contain, wanted or not. Car is weakened slightly, because the nonrelevant review is mostly car.
The cosines of the documents with the original and revised queries:
| Document | With $\vec{q}_0$ | With $\vec{q}_m$ |
|---|---|---|
| $d_1$ | 0.686 | 0.802 |
| $d_2$ | 0.425 | 0.348 |
| $d_3$ | 0.516 | 0.671 |
| $d_4$ | 0.086 | 0.117 |
Both relevant documents gain and the nonrelevant review loses, which is what the update was built to do. The car-loan page gains a little through rate. The ranking itself does not change here, since it was already right, but the gap between the relevant and nonrelevant documents has widened from 0.091 (0.516 against 0.425) to 0.323 (0.671 against 0.348). On a real collection that margin is what lets documents using auto, and never car, climb into the results.
Relevance feedback works well when the relevant documents resemble one another and differ from the nonrelevant ones. It fails when relevant documents fall into several distinct groups, or when the initial query retrieves nothing relevant to learn from. Searchers are also often unwilling to make the judgments. Pseudo-relevance feedback avoids that by assuming the top few results are relevant. Relevance feedback and pseudo-relevance feedback have their own articles, which extend this treatment to probabilistic models.
Descendants
The model's ideas did not disappear when other models surpassed it. They were reworked.
Latent semantic indexing (Deerwester et al., 1990) attacks the independence assumption directly. It factors the term-document matrix into a small number of latent dimensions, so that terms which occur in similar documents, such as car and automobile, are mapped close together. Queries and documents are compared by cosine in that reduced space. It is the first of the methods in topic modeling.
BM25 keeps the vector space model's ingredients, a dampened term frequency, an IDF, and a tunable length normalization, but motivates their form with a probabilistic model (see Okapi BM25). In many respects it is a pivoted, saturated TF-IDF with a theory behind it.
Dense retrieval returns to vectors and cosines, but with learned representations. A neural encoder maps queries and documents to vectors of several hundred to a few thousand dimensions in which related meanings lie close together. Retrieval is then nearest-neighbor search by dot product or cosine (see the history). The geometry is the vector space model's. What changed is that the dimensions are learned from data rather than defined as words, which addresses the vocabulary mismatch that orthogonal term dimensions could not.
The running-example numbers are computed by tools/experiments/vsm/running_example.py in the site repository, and the figure by tools/figures/vector-space-model-limits.py.
References
- Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., and Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6), 391–407.
- Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, chapter 6 and section 9.1. Cambridge University Press.
- Raghavan, V. V., and Wong, S. K. M. (1986). A critical analysis of vector space model for information retrieval. Journal of the American Society for Information Science, 37(5), 279–287.
- Rocchio, J. J. (1971). Relevance feedback in information retrieval. In G. Salton (Ed.), The SMART Retrieval System: Experiments in Automatic Document Processing, 313–323. Prentice-Hall.