Every retrieval model so far has treated words as independent dimensions. A document about automobiles and a query about cars share no terms, so the vector space model scores them zero, however closely they are related. A document about jaguars the animal and one about Jaguar the car share a term, so they look alike. Words are an imperfect stand-in for what documents are about.
Topic models represent documents by what they are about. They find, from word co-occurrence across a whole collection, a small number of latent topics, each a pattern of words that tend to appear together, and describe each document as a mixture of those topics. No one specifies the topics in advance. Like clustering, topic modeling is unsupervised, but where clustering puts each document in one group, a topic model lets a document belong partly to several: a story about an oil company's takeover bid is partly about oil and partly about acquisitions.
The topic follows the historical sequence. Latent semantic indexing (1990) used linear algebra, factoring the term-document matrix with the singular value decomposition. Probabilistic latent semantic analysis (1999) recast the idea as a statistical model. Latent Dirichlet allocation (2003) made that model fully generative, and became the standard method for exploring large text collections. Measuring whether a topic model is any good turns out to be as hard as fitting one.
The course ran out of time before reaching this topic, so these articles were written from the textbook and the original papers rather than from lectures. The experiments test the methods on the collections used in earlier topics: LSI for retrieval on Cranfield, and LDA, LSI, and nonnegative matrix factorization for topic discovery on Reuters. Alongside them, one tiny collection runs through every article, the textbook's six documents over the words ship, boat, ocean, voyage, and trip, small enough that each method can be worked by hand: its SVD computed in full, two iterations of pLSA's EM traced, and LDA's Gibbs sampler stepped word by word and checked against the exact posterior.
Articles
The idea
- What is topic modeling: topics as distributions over words, documents as mixtures of topics, how topic models relate to clustering, preparing text for them, and what they are used for.
The models
- Latent semantic indexing: the singular value decomposition built from an eigendecomposition, low-rank approximation, the textbook's worked example computed step by step, folding in queries, and an experiment on Cranfield in which LSI outperformed BM25.
- Probabilistic latent semantic analysis: the aspect model and its geometry, expectation-maximization derived and run by hand, and why it overfits and has no model for new documents.
- Latent Dirichlet allocation: the generative process and plate notation, Dirichlet priors, collapsed Gibbs sampling derived, traced, and explored in an interactive demo, variational inference, and twenty topics found in Reuters.
Judging and extending
- Evaluating topic models: held-out perplexity derived and computed, human judgments by word and topic intrusion, coherence measures worked by hand, choosing the number of topics, and where these disagree on Reuters.
- Topic models beyond LDA: topic models in retrieval, correlated, dynamic, supervised, and nonparametric models, nonnegative matrix factorization, and neural and embedding-based topic models.
Related topics
- Limits of the vector space model describes the vocabulary problem that LSI was designed to address.
- Text clustering solves the hard version of the same problem, one group per document.
- The 2-Poisson model and K-means introduced the mixture models and expectation-maximization that pLSA and LDA build on.
Course materials
The course did not reach this topic, and there was no assignment for it. The experiments were written for this site and are in tools/experiments/topics/ in the site repository.
References
- Blei, D. M. (2012). Probabilistic topic models. Communications of the ACM, 55(4), 77–84.
- Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.
- Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., and Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6), 391–407.
- Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, chapter 18. Cambridge University Press.