Overview
An introduction to indexing and information retrieval systems, covering text processing, Boolean search, the vector space model, IR system evaluation, probabilistic search, text classification, clustering, and web crawling.
A few sentences on what the course was like and what you focused on.
Topics
The course followed the sequence of Manning, Raghavan, and Schütze's Introduction to Information Retrieval, building a working search engine one layer at a time. I am rewriting my study notes as a page for each topic, each one leading to detailed articles on its concepts. Topics without a link are still in progress.
- Foundations. What retrieval systems do and how a search engine is put together.
- Text processing and indexing. Tokenization, normalization, stemming, and the inverted index.
- Boolean retrieval. Posting list intersection, phrase queries, and query optimization.
- The vector space model. TF-IDF weighting and cosine similarity.
- Evaluation. Precision, recall, MAP, and nDCG.
- Probabilistic retrieval. The Probability Ranking Principle, the Binary Independence Model, and BM25.
- Text classification. Naive Bayes and feature selection.
- Clustering. K-means and cluster evaluation.
- Web crawling. Crawler architecture, politeness, freshness, and PageRank.
- Topic modeling. LSI, pLSA, LDA, and evaluating topics.
Projects
Sentiment and Search over Goodreads Reviews
Using the UCSD Book Graph dataset (Wan and McAuley, 2018), we built a Python system with a PyQt6 tool that loads Goodreads data into SQLite and a pipeline that indexes 1.26 million reviews (540,135 distinct terms) for search with the Binary Independence Model. On top of the index we trained a multinomial Naive Bayes classifier to predict a review's star rating from its text. On 120,000 reviews it chose the correct rating 61% of the time, against 36% for always guessing the most common rating, and 80% of its errors were off by a single star. A K-means analysis of reviewers separated them into groups that differ mainly in review length, and the most verbose reviewers gave the lowest average ratings.
Your role in the project and what you would do differently now.
Homework: Building a Search Engine
Four assignments built a search engine from the ground up, and the Goodreads project reused their code:
- HW01: an inverted index with tokenization and stemming.
- HW02: Boolean query parsing and processing.
- HW03: document similarity in the vector space model, with TF-IDF and sublinear weighting and parallel implementations.
- HW04: probabilistic ranking with the Binary Independence Model.
Takeaways
What you learned and where you have applied it since.