Topic Models Beyond LDA

Latent Dirichlet allocation became less a single model than a template. Its generative story can be changed piece by piece: add time, add authors, add labels, relax the assumption that topics are unrelated, or let the data choose how many topics there are. Each change gives a new model with its own inference procedure. Separately, methods from outside the Bayesian tradition, matrix factorization and later neural networks and pretrained language models, came to compete with LDA at its main job. This article surveys both lines and returns to the question topic modeling started from, whether it improves retrieval.

Topic models in retrieval

LSI was proposed as a retrieval model, and on Cranfield it beat BM25. LDA was applied to retrieval differently, as a way of smoothing a document's language model. In the query likelihood approach, a document is ranked by the probability that a model of its words would generate the query, and that model must give some probability to words the document does not contain. Wei and Croft (2006) mixed the usual estimate, the document's word frequencies smoothed with the collection's, with a topic-based estimate:

$$P(w \mid d) = \lambda\, P_{\text{doc}}(w \mid d) + (1 - \lambda) \sum_{k} P(w \mid z = k)\, P(z = k \mid d).$$

The second term lets a document about oil exploration give some probability to drilling even if it never uses the word, because its topics do.

The six-document example of the LSI and pLSA articles shows the effect in miniature. Take the one-word query boat, the two pLSA topics (sea: ship 0.4, boat 0.2, ocean 0.4; travel: voyage 0.6, trip 0.4), and, for illustration, $\lambda = 0.7$ with an unsmoothed document estimate:

The document containing boat still ranks first, by far, because the exact-match term carries most of the weight. Behind it, the topic term separates the documents that lack the word: $d_3$, entirely about the sea, gets more than $d_1$, which is one-third travel, and the travel documents get nothing. With word matching alone, those five documents would tie at zero. The values are from beyond_toy.py in tools/experiments/topics/.

Wei and Croft reported improvements over query likelihood alone on all five TREC collections they tested, with effectiveness close to that of relevance models, the probabilistic form of pseudo-relevance feedback, on most of them. The cost is fitting a topic model to the whole collection, and the topic term must be combined with exact matching, not substituted for it. Topic models have since been used more for exploring collections than for ranking, and dense neural representations took over the role of matching beyond shared words.

Extensions of LDA

• Correlated topic models (Blei and Lafferty, 2007). LDA's Dirichlet prior cannot express that some topics tend to appear together, such as genetics and disease. Replacing it with a logistic normal distribution, which has a covariance matrix, lets the model learn which topics co-occur.
• Dynamic topic models (Blei and Lafferty, 2006). A topic's vocabulary changes over time. The dynamic model lets each topic's word distribution drift from one time period to the next, so that a physics topic can be traced through a century of a journal's articles as its characteristic words change.
• Supervised LDA (Blei and McAuliffe, 2007). Each document has a response, such as a rating, and the topics are fitted to predict it as well as to explain the words. Labeled LDA (Ramage, Hall, Nallapati, and Manning, 2009) instead ties topics to the tags people have given documents.
• Author-topic models (Rosen-Zvi, Griffiths, Steyvers, and Smyth, 2004). Topics are attached to authors, and a document's words come from the topics of its authors, which lets the model describe what each author writes about.
• Structural topic models (Roberts and colleagues, 2014). Document metadata, such as the date, the source, or a survey respondent's party, can shift both topic prevalence and topic vocabulary. Structural topic models are widely used in political science.

Choosing the number of topics automatically

Every model above needs $K$ in advance. The hierarchical Dirichlet process (Teh, Jordan, Beal, and Blei, 2006) allows an unbounded number of topics, with a prior under which only finitely many are used in any finite collection, and infers how many the data support. The approach is called nonparametric, though it has concentration parameters of its own that influence how many topics it finds. As evaluating topic models explains, the right number of topics depends on the purpose as well as the data, so automatic selection removes a choice without removing the need to judge the result.

Nonnegative matrix factorization

Nonnegative matrix factorization (NMF) factors the document-term matrix $X$, of size documents by terms, into two nonnegative matrices, $X \approx WH$, where each row of $H$ is a topic's weights over terms and each row of $W$ is a document's weights over topics (Lee and Seung, 1999). Like LSI, it is a low-rank approximation. Unlike LSI, it allows no negative values, so a document can only be built up by adding topics, never by subtracting them, and the factors read as parts. With the Kullback–Leibler divergence as its measure of error, NMF is closely related to pLSA.

NMF on the six-document example

Factoring the example's $5 \times 6$ term-document count matrix (here terms by documents, so the term factor comes first) with two components shows both the parts-based reading and the role of the error measure. With the generalized KL divergence (multiplicative updates, best of 20 random starts), the term factor is

and the document factor gives $d_1$ weights 1.078 and 0.390, $d_2$ 1.078 and 0, $d_3$ 0.539 and 0, and the travel documents weight only on component 2. Every entry is nonnegative, and the components are the sea and travel word groups. Dividing each component's term weights by their sum (0.742 + 0.371 + 0.742 = 1.855 for the first, 1.537 + 1.025 = 2.562 for the second) turns them into $P(w \mid z)$ = (0.4, 0.2, 0.4) and (0.6, 0.4). Multiplying the document weights by the same sums gives expected counts, $d_1$'s being $1.078 \times 1.855 = 2.0$ and $0.390 \times 2.562 = 1.0$, so its mixture is (0.667, 0.333): exactly the best pLSA solution, with the same log-likelihood, −10.549, as Gaussier and Goutte (2005) predict.

With squared error instead, the solution changes. The first component becomes ship 0.63, boat 0.28, ocean 0.73, and voyage 0.36, absorbing part of voyage to fit $d_1$ better, and the Frobenius error is 1.736. That is close to, but above, the 1.668 of LSI's rank-2 approximation, which by the Eckart–Young theorem no rank-2 matrix can beat; the difference, 0.068, is the cost of requiring every entry to be nonnegative.

NMF on Reuters

NMF is fast, deterministic with a fixed initialization, and often gives readable topics. On the same Reuters training stories as the LDA experiment, NMF with 20 topics, applied to TF-IDF weights with scikit-learn's default squared-error objective rather than the KL divergence, reached a mean NPMI coherence of 0.288, against 0.251 for LDA, and its least coherent topic scored 0.020 against LDA's −0.329.

The comparison has a catch that coherence does not show. About half of NMF's twenty topics described the formats of earnings reports: dividend notices, tables of revenue and share counts, notes on discontinued operations, half-year figures, and so on. Several are highly coherent, because those notices use fixed phrases; the dividend topic scored 0.476. NMF had one topic for acquisitions where LDA had three, and none for coffee and cocoa, which LDA found. Coherence scores each topic alone, so it can reward narrow, formulaic topics and does not penalize a model for leaving subjects out. Judging a model also requires asking what it covers.

Neural and embedding-based topic models

• Neural variational topic models. ProdLDA (Srivastava and Sutton, 2017) fits an LDA-like model with a neural network that maps a document to its topic proportions, trained by variational autoencoding. Inference for a new document is a single pass through the network. The embedded topic model (Dieng, Ruiz, and Blei, 2020) represents words and topics as vectors in the same space, so that topics can use words' meanings, learned from word embeddings, as well as their co-occurrence in the collection.
• Clustering document embeddings. BERTopic (Grootendorst, 2022) takes a different route. It encodes each document with a pretrained sentence-embedding model, reduces the dimensionality, clusters the documents, and describes each cluster by the words that distinguish it, using a class-based version of TF-IDF. Because the embeddings come from models trained on enormous amounts of text, documents can be grouped by meaning even when they share few words. Each document is assigned to one cluster or left out as an outlier, so this is closer to clustering than to a mixture model, though topic distributions can be approximated afterward.
• Language models as topic modelers. Large language models can read documents, propose topics in plain language, and assign documents to them. TopicGPT (Pham, Hoyle, Sun, Resnik, and Iyyer, 2024) prompts a language model to generate and refine a list of topics with descriptions, then to label each document. The topics are readable sentences rather than word lists, at the cost of running a large model over every document and of results that depend on the prompt.

Embedding-based methods are often preferred for short or informal text, where word co-occurrence is sparse, though, as Hoyle and colleagues (2021) caution, claims of better topics need checking with people. LDA remains widely used where its properties matter: a clear probabilistic model, mixed membership, and results that can be reproduced and explained.

The NMF comparison is reuters_nmf.py, and the small examples are beyond_toy.py, in tools/experiments/topics/ in the site repository.

References

• Blei, D. M., and Lafferty, J. D. (2006). Dynamic topic models. In Proceedings of ICML 2006, 113–120.
• Blei, D. M., and Lafferty, J. D. (2007). A correlated topic model of Science. The Annals of Applied Statistics, 1(1), 17–35.
• Blei, D. M., and McAuliffe, J. D. (2007). Supervised topic models. In Advances in Neural Information Processing Systems 20 (NIPS 2007), 121–128.
• Dieng, A. B., Ruiz, F. J. R., and Blei, D. M. (2020). Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8, 439–453.
• Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794.
• Hoyle, A., Goel, P., Hian-Cheong, A., Peskov, D., Boyd-Graber, J., and Resnik, P. (2021). Is automated topic model evaluation broken? The incoherence of coherence. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021).
• Lee, D. D., and Seung, H. S. (1999). Learning the parts of objects by non-negative matrix factorization. Nature, 401, 788–791.
• Pham, C. M., Hoyle, A., Sun, S., Resnik, P., and Iyyer, M. (2024). TopicGPT: A prompt-based topic modeling framework. In Proceedings of NAACL 2024, 2956–2984.
• Ramage, D., Hall, D., Nallapati, R., and Manning, C. D. (2009). Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of EMNLP 2009, 248–256.
• Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., and Rand, D. G. (2014). Structural topic models for open-ended survey responses. American Journal of Political Science, 58(4), 1064–1082.
• Rosen-Zvi, M., Griffiths, T., Steyvers, M., and Smyth, P. (2004). The author-topic model for authors and documents. In Proceedings of UAI 2004, 487–494.
• Srivastava, A., and Sutton, C. (2017). Autoencoding variational inference for topic models. In Proceedings of ICLR 2017.
• Teh, Y. W., Jordan, M. I., Beal, M. J., and Blei, D. M. (2006). Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476), 1566–1581.
• Wei, X., and Croft, W. B. (2006). LDA-based document models for ad-hoc retrieval. In Proceedings of SIGIR 2006, 178–185.