A collection of news stories contains thousands of distinct words, but far fewer subjects. Stories about interest rates use bank, rate, money, central, and fed together. Stories about takeovers use offer, shares, merger, tender, and bid. A topic model finds these recurring groups of words automatically, from nothing but the documents themselves, and describes each document by how much of each group it contains. This article explains what that means and why it is useful before the models themselves are introduced.
The problem topic models address
The retrieval models of earlier topics match words. Two failures follow from that, and both motivated latent semantic indexing (Deerwester, Dumais, Furnas, Landauer, and Harshman, 1990; Manning, Raghavan, and Schütze, 2008, §18.4):
- Synonymy. Different words can mean the same thing. A query for car misses documents that say only automobile, and a searcher's word for a concept often differs from an author's. Furnas, Landauer, Gomez, and Dumais (1987) found that two people spontaneously chose the same word for the same thing less than 20 percent of the time.
- Polysemy. The same word can mean different things. A query for jaguar matches documents about the animal, the car, and the operating system, which share the word but nothing else.
The limits of the vector space model traced both to its treatment of every term as an independent dimension. The remedy that topic models offer is to find dimensions that correspond to meaning. If car and automobile both belong to the same topic, documents using either word are close on that topic. If jaguar belongs partly to an animal topic and partly to a car topic, the other words in a document decide which applies.
Topics and mixtures
The probabilistic topic models share a simple picture (Blei, 2012):
- A topic is a probability distribution over the vocabulary. An interest-rate topic gives high probability to bank, rate, and money and almost none to wheat. Every topic assigns some probability to every word, but in practice each is characterized by its few dozen most probable words.
- A document is a mixture of topics. A story might be 60 percent interest rates, 30 percent currency markets, and 10 percent other subjects. Its words are imagined as drawn from those topics in those proportions.
Fitting a topic model to a collection means working backward: given only the words of the documents, find topics and mixtures that would plausibly have produced them. The topics are latent, never observed directly, and inferred from the way words co-occur across many documents. Words that often appear in the same documents end up in the same topics.
A small example
The articles in this topic share one tiny collection, taken from Manning, Raghavan, and Schütze (2008, §18.4): six documents over five words, $d_1$ = ship ocean voyage, $d_2$ = boat ocean, $d_3$ = ship, $d_4$ = voyage trip, $d_5$ = voyage, and $d_6$ = trip. Two topics describe it well. A sea topic gives ship and ocean probability 0.4 each and boat 0.2. A travel topic gives voyage 0.6 and trip 0.4. Documents $d_2$ and $d_3$ are entirely sea, $d_4$ to $d_6$ entirely travel, and $d_1$ is two-thirds sea and one-third travel. These are the values that probabilistic latent semantic analysis finds when fitted to the six documents.
The mixture picture says that each document's word distribution is its topics' distributions averaged with its own weights. For $d_1$:
$$P(\text{ship} \mid d_1) = \tfrac{2}{3} \times 0.4 + \tfrac{1}{3} \times 0 = 0.267, \qquad P(\text{voyage} \mid d_1) = \tfrac{2}{3} \times 0 + \tfrac{1}{3} \times 0.6 = 0.2.$$
Doing this for every word and document is a matrix product: a words-by-documents table of probabilities equals a words-by-topics table times a topics-by-documents table.
The model is not a copy of the data. $d_3$ contains only ship, but the model gives it a 0.2 chance of boat, because it is a sea document and sea documents use boat. That is how a topic model handles synonymy: a query for boat can match $d_3$ through the sea topic, though the word never appears in it.
Latent semantic indexing reaches a related representation by linear algebra instead of probability. Its dimensions are not distributions, and a document's coordinates on them can be negative, but the purpose is the same: a few dimensions, learned from co-occurrence, that capture what documents are about better than individual words do.
How the models compare
Every model in this topic factors the term-document matrix into a small number of dimensions. They differ in what the factors may contain and in what they optimize:
| Model | Factors | Entries | Fitted by | New documents |
|---|---|---|---|---|
| LSI (1990) | term patterns × weights × document patterns | any sign | least squares (SVD) | folded in by projection |
| pLSA (1999) | $P(w \mid z)$ × $P(z \mid d)$ | probabilities | maximum likelihood (EM) | folded in by re-running EM |
| LDA (2003) | $\phi$ × $\theta$, with Dirichlet priors | probabilities | Bayesian inference (sampling or variational) | have a probability under the model |
| NMF (beyond LDA) | nonnegative × nonnegative | nonnegative weights | least squares or KL divergence | fitted with the topics fixed |
Reading down the table is reading the history: each model kept its predecessor's factorization and changed what was wrong with it.
Topic models, clustering, and classification
Topic modeling is unsupervised, like clustering. No categories or labeled examples are given, and people must interpret what is found. The difference is in what a document is allowed to be. K-means assigns each document to exactly one cluster. A topic model gives each document a degree of membership in every topic, a soft or mixed-membership assignment. That suits documents, which often cover several subjects, and it also lets the same word belong to several topics, which handles polysemy.
The topics also differ from the categories of classification. Someone defined the Reuters categories with a purpose. A topic model finds whatever patterns of co-occurrence are strongest in the collection, and those may cut across the categories people would choose. In the Reuters experiment described in latent Dirichlet allocation, the single category of earnings reports was divided among several topics, among them results tables, dividend announcements, stock splits, and the language of quarterly earnings.
Preparing text
The probabilistic models work with word counts, so documents are represented as bags of words, ignoring order. Preparation follows the text processing pipeline with a few differences in emphasis:
- Stop words matter more. Words such as the and said appear in nearly every document. A topic model will otherwise spread them across all topics or gather them into a topic of their own. Collections usually need additional, collection-specific stop words. For Reuters, abbreviations such as mln, dlrs, and pct were removed as well.
- Very rare and very common words are usually dropped. A word in two documents gives too little evidence of a co-occurrence pattern, and a word in half of them distinguishes nothing.
- Stemming is often skipped. It reduces the vocabulary, but stemmed words make topics harder to read, and readability is the point of many applications.
- Weighting depends on the model. LSI is usually applied to weighted vectors, such as TF-IDF. LDA and pLSA model raw counts, since their probabilities are defined over word occurrences.
What topic models are used for
- Exploring a collection. The commonest use is to get an overview of a collection too large to read: which subjects it covers, in what proportions, and how they change over time. Historians, political scientists, and literary scholars have used topic models to study newspaper archives, legislative speeches, and scientific literatures.
- Retrieval. LSI was proposed as a retrieval model, matching queries and documents in the latent space. Topic models have also been used to smooth document language models, so that a document's model gives some probability to words it does not contain but its topics do, as in topic models beyond LDA.
- Features for other tasks. A document's topic proportions are a short, dense description of it, useful as input to classifiers, for finding similar documents, and for recommendation.
- Organizing and browsing. Topics can label and connect documents in an interface, as clusters can, with the difference that a document can appear under several topics.
References
- Blei, D. M. (2012). Probabilistic topic models. Communications of the ACM, 55(4), 77–84.
- Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., and Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6), 391–407.
- Furnas, G. W., Landauer, T. K., Gomez, L. M., and Dumais, S. T. (1987). The vocabulary problem in human-system communication. Communications of the ACM, 30(11), 964–971.
- Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, chapter 18. Cambridge University Press.