For our term project in CSC 790, James R. Brown and I built a system that searches Goodreads reviews, classifies them by rating, and clusters the people who write them. This article is about the classification part. The task was to predict a review's star rating, from one to five, from its text alone, a five-class text classification problem and a form of sentiment analysis. The code is in the Goodreads repository. Looking back at the project after writing the rest of this topic, I can describe what it showed more precisely than we did at the time.
Data and pipeline
The reviews come from the UCSD Book Graph, a large collection of Goodreads data gathered for research on recommendation and review analysis (Wan and McAuley, 2018; Wan et al., 2019). The project had two programs. A desktop application, written with a PyQt6 interface, downloaded the data and loaded it into SQLite. A command-line pipeline built an inverted index over 1.26 million reviews and 540,135 distinct terms, searched it with the Binary Independence Model, and ran the classification and clustering phases. For classification we used a set of 119,997 reviews and their ratings.
Each review was lowercased, tokenized with NLTK, stripped of non-letter characters, filtered to remove one-letter tokens, and stemmed with the Porter stemmer. Stop words were not removed in this path. The classifier is multinomial Naive Bayes with add-one smoothing, implemented directly from the algorithm in Manning, Raghavan, and Schütze (2008, §13.2). We wrote code for feature selection by frequency, χ², and mutual information, the last only as a rough sketch, but the final runs used the full vocabulary.
Results
The overall figures we reported:
| Measure | Value |
|---|---|
| Accuracy | 0.607 |
| Macro-averaged precision | 0.636 |
| Macro-averaged recall | 0.495 |
| Macro-averaged F1 | 0.517 |
The confusion matrix shows what lies behind them. Reading it class by class:
| True rating | Share of reviews | Recall | Precision |
|---|---|---|---|
| 1 star | 3.4% | 0.16 | 0.76 |
| 2 stars | 8.6% | 0.35 | 0.61 |
| 3 stars | 22.1% | 0.67 | 0.52 |
| 4 stars | 36.1% | 0.59 | 0.62 |
| 5 stars | 29.9% | 0.71 | 0.67 |
Three patterns stand out.
- Performance follows class frequency. The rare ratings are the weak ones. One-star reviews, 3.4 percent of the data, were recognized only 16 percent of the time. The classifier predicted one star just 850 times in nearly 120,000 reviews, though when it did, it was right 76 percent of the time. Naive Bayes's priors and the small amount of training text for rare classes both pull predictions toward common ratings.
- Three stars is a catch-all. The classifier predicted three stars 34,251 times, against 26,520 reviews that actually had three stars. Mixed and uncertain reviews tended to land there.
- Most errors are near misses. Of the 47,112 errors, 80 percent were off by a single star, and the mean absolute error across all reviews was 0.49 stars. The classifier rarely confused a positive review with a negative one. It mostly confused neighboring ratings.
The right comparison
The natural comparison point is the majority class (see evaluating classifiers). Four-star reviews are the most common, at 36.1 percent, so a classifier that always predicted four stars would be 36.1 percent accurate. At 60.7 percent, ours learned a good deal beyond the class distribution.
Next steps
- Treat the ratings as ordered. Mean absolute error, 0.49 stars here, describes this classifier better than accuracy, because it credits near misses. Regression or ordinal classification methods would use the order directly.
- Address the imbalance. Adjusting the priors or rebalancing the training data would show how much of the poor recall on one- and two-star reviews comes from their rarity.
- Compare stronger classifiers. On Reuters, k nearest neighbors beat Naive Bayes. Logistic regression or a support vector machine would be the natural next comparison, and a fine-tuned language model the current standard for sentiment.
- Use the feature selection code. Comparing the full vocabulary with a few thousand χ² or frequency-selected terms would show whether the classifier needs all of them, as feature selection did on Reuters.
What the project showed holds up. A probabilistic classifier written from scratch learned enough from review text to get the rating right well above the baseline, and when it was wrong it was usually wrong by a single star.
References
- Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, chapter 13. Cambridge University Press.
- Wan, M., and McAuley, J. (2018). Item recommendation on monotonic behavior chains. In Proceedings of RecSys 2018, 86–94.
- Wan, M., Misra, R., Nakashole, N., and McAuley, J. (2019). Fine-grained spoiler detection from large-scale review corpora. In Proceedings of ACL 2019, 2605–2610.