Designing a Modular Search Engine

A search engine built over the course of a semester changes constantly. The first version answers Boolean queries. A few weeks later it needs TF-IDF weighting and cosine similarity, then probabilistic weights, then BM25, and eventually classifiers and clustering built on the same index. Each new model has to be compared with the previous ones on the same documents and queries, which is only a fair comparison if everything except the model stays fixed. A system written as one long script makes this painful: changing the ranking function means editing code that also tokenizes, indexes, and prints results, and every change risks breaking something unrelated.

The remedy is a modular design, in which each stage of the search engine pipeline is a separate component behind an interface, so that any one of them can be replaced without touching the others. This article describes such a design, gives a working sketch in Python, and draws lessons from the code I wrote for the course, which applied these ideas, sometimes imperfectly.

Why modularity matters for retrieval in particular

Modularity is good general practice, but retrieval has three specific reasons to insist on it.

Experiments compare one component at a time. Retrieval research advances by controlled comparison: hold the collection, the queries, the relevance judgments, and the text analysis fixed, change one thing, and measure the difference. If the ranking function cannot be swapped without also changing the tokenizer, a difference in effectiveness cannot be attributed to the ranking function. The experimental method described in evaluation requires a system whose parts can be varied independently.

Documents and queries must be analyzed identically. As the architecture article showed, a query must pass through exactly the same tokenization, stop-word removal, and stemming as the documents, or terms silently fail to match. Deliberate query-side additions, such as synonym expansion, are a separate design decision; the danger is accidental differences. The most reliable way to guarantee that is to have one analysis component that both paths call. Duplicating the analysis code in two places invites the two copies to drift apart.

The index is a contract. Each ranking model needs particular statistics from the index: postings, term frequencies, document frequencies, document lengths. If ranking code reads the index's internal data structures directly, then changing how the index stores postings, for instance by compressing them or moving them to disk, breaks every ranking model at once. If ranking code asks the index for statistics through a defined interface, the index can change freely behind it.

The components

A small search engine divides naturally into the following components. Each corresponds to a stage of the pipeline and to a later topic in the course.

The dependencies run in one direction. The index depends on the analyzer. The ranker depends on the index interface, not on any particular index. The search engine connects an analyzer, an index, and a ranker, and the evaluator depends only on the search engine's ability to answer queries. Nothing depends on the presenter, so the same engine can drive a command-line interface, a web page, or an evaluation script.

A working sketch

The following Python code implements the core of this design. It is deliberately small, but it runs, and it reproduces the four-document example from the architecture article.

The analyzer is a single class used for both documents and queries:

import math
import re
from abc import ABC, abstractmethod
from collections import Counter, defaultdict

from nltk.stem import PorterStemmer


class Analyzer:
    """Turns raw text into index terms. Used for documents and queries alike."""

    def __init__(self, stop_words):
        self.stop_words = set(stop_words)
        self.stemmer = PorterStemmer(mode=PorterStemmer.ORIGINAL_ALGORITHM)

    def analyze(self, text):
        tokens = re.findall(r"[a-z]+", text.lower())
        return [self.stemmer.stem(t) for t in tokens if t not in self.stop_words]

The index interface states what a ranking model may ask of an index, and nothing more. An in-memory implementation stores postings in dictionaries. A later implementation could store compressed postings on disk, and no ranking code would change.

class Index(ABC):
    """What a ranking model may ask of an index, and nothing more."""

    @abstractmethod
    def postings(self, term):
        """Return a list of (doc_id, term_frequency) pairs."""

    @abstractmethod
    def doc_freq(self, term): ...

    @abstractmethod
    def doc_length(self, doc_id): ...

    @property
    @abstractmethod
    def num_docs(self): ...


class InMemoryIndex(Index):
    def __init__(self, analyzer):
        self.analyzer = analyzer
        self._postings = defaultdict(dict)
        self._lengths = {}

    def add(self, doc_id, text):
        if doc_id in self._lengths:
            raise ValueError(f"document already indexed: {doc_id}")
        terms = self.analyzer.analyze(text)
        self._lengths[doc_id] = len(terms)
        for term, tf in Counter(terms).items():
            self._postings[term][doc_id] = tf

    def postings(self, term):
        return sorted(self._postings.get(term, {}).items())

    def doc_freq(self, term):
        return len(self._postings.get(term, {}))

    def doc_length(self, doc_id):
        return self._lengths[doc_id]

    @property
    def num_docs(self):
        return len(self._lengths)

The sketch rejects a second add with an identifier already in the index. Simply overwriting would leave the old version's postings in place, so that a replaced document would still match terms it no longer contains. Supporting updates properly requires a delete operation that removes those postings and adjusts the collection statistics.

Rankers share one method. A Boolean conjunction and a TF-IDF scorer are interchangeable because both use only the index interface:

class Ranker(ABC):
    @abstractmethod
    def score(self, query_terms, index):
        """Return {doc_id: score} for documents matching the query."""


class BooleanAnd(Ranker):
    def score(self, query_terms, index):
        sets = [{d for d, _ in index.postings(t)} for t in query_terms]
        return {d: 1.0 for d in set.intersection(*sets)} if sets else {}


class TfIdf(Ranker):
    def score(self, query_terms, index):
        scores = defaultdict(float)
        for t in query_terms:
            df = index.doc_freq(t)
            if df == 0:
                continue
            idf = math.log10(index.num_docs / df)
            for doc_id, tf in index.postings(t):
                scores[doc_id] += tf * idf
        return dict(scores)

Because TfIdf loops over the analyzed query terms, a term that appears twice in the query contributes twice to each score, which amounts to weighting query terms by their frequency in the query. Iterating over set(query_terms) would treat the query as a set instead.

The search engine connects the parts. It analyzes the query with the same analyzer the index used, asks the ranker for scores, and sorts them. Ties are broken by document identifier, so the output is deterministic:

class SearchEngine:
    def __init__(self, analyzer, index, ranker):
        self.analyzer, self.index, self.ranker = analyzer, index, ranker

    def search(self, query, k=10):
        terms = self.analyzer.analyze(query)
        scores = self.ranker.score(terms, self.index)
        return sorted(scores.items(), key=lambda item: (-item[1], item[0]))[:k]

Nothing in these classes checks that the analyzer given to SearchEngine is the one the index was built with. Both would satisfy their interfaces, and a mismatch would only lower effectiveness without raising an error. A larger system records the analyzer configuration with the index and builds the query analyzer from it.

Running both rankers on the example collection:

docs = {
    "d1": "Arctic sea ice is declining as the climate warms.",
    "d2": "A warming climate threatens Arctic seabirds and their food.",
    "d3": "Climate models predict warmer winters in Europe.",
    "d4": "Seabirds of the Arctic: a field guide.",
}
analyzer = Analyzer(["a", "and", "as", "in", "is", "of", "the", "their"])
index = InMemoryIndex(analyzer)
for doc_id, text in docs.items():
    index.add(doc_id, text)

for ranker in (BooleanAnd(), TfIdf()):
    engine = SearchEngine(analyzer, index, ranker)
    print(type(ranker).__name__, engine.search("arctic climate warming"))

produces, with scores rounded to three places,

BooleanAnd [('d1', 1.0), ('d2', 1.0)]
TfIdf [('d1', 0.551), ('d2', 0.551), ('d3', 0.125), ('d4', 0.125)]

which matches the hand computation in the architecture article. Adding BM25 later means writing one new Ranker subclass. It also needs the average document length, which the interface above does not provide: a ranker cannot sum doc_length without a way to list the documents. The fix is to add one method, such as avg_doc_length, to Index and implement it once in each index class. That is the normal way an interface grows. A new model reveals a statistic it needs, the index learns to supply it, and the analyzer and search engine are unchanged.

Testing

A modular design makes retrieval code testable, and retrieval code needs tests more than it might appear. A bug in a ranking function rarely causes an error. It produces a slightly different ranking that looks plausible. The most useful tests are therefore small collections with hand-computed expected scores, such as the four-document example:

def test_tfidf_matches_hand_computation():
    engine = SearchEngine(analyzer, index, TfIdf())
    results = dict(engine.search("arctic climate warming"))
    assert abs(results["d1"] - 0.551) < 1e-3
    assert abs(results["d3"] - 0.125) < 1e-3

Other useful tests check properties that should hold for any correct implementation: that a document containing none of the query terms is never returned, that the analyzer produces the same terms for a word whether it appears in a document or a query, that adding a document identifier twice is rejected without changing the index, that saving and loading an index, for implementations that support it, leaves every statistic unchanged, and that two index implementations return identical postings for the same collection. The last kind of test is what makes it safe to replace a simple index with a faster one.

Beyond unit tests, the evaluator provides the end-to-end test. Given a set of queries with relevance judgments, it computes measures such as precision and average precision for each ranker. Whenever a component changes, rerunning the evaluation shows whether effectiveness improved, worsened, or stayed the same.

Lessons from my course code

My homework solutions for the course, in the information-retrieval repository, and the Goodreads project that grew out of them, applied this kind of design, and reviewing them afterward shows both what worked and what I would change.

Factories for choosing implementations. HW03 defines an abstract BaseIndex with two implementations, a standard single-process index and a parallel one that uses multiple processes to build large indexes faster. An IndexFactory chooses between them, selecting the parallel index when the collection has at least 5,000 documents and multiprocessing is available, and falling back to the standard index otherwise. The vector space models follow the same pattern, with standard, sparse-matrix, and parallel implementations behind one interface. The calling code never needs to know which implementation it received, which is the main benefit the design promises.

Consistent analysis, with a coupling cost. In the Goodreads project, the Binary Independence Model processes queries by calling the index's own preprocessing method, so queries and documents are guaranteed to be analyzed identically. That was the right goal. But the method is a private one (_preprocess_text), and to find candidate documents the BIM code reads the index's internal dictionary of postings (term_doc_freqs) directly. That dictionary is not part of BaseIndex. An index implementation that stored postings differently would satisfy the interface and still break the BIM. The fix is the structure in the sketch above: make the analyzer a separate component that both the index and the query path use, and give the index interface a postings method so rankers never touch its internals.

Interfaces should be narrow. The BaseIndex in HW03 declares fourteen abstract methods and properties, including JSON export, memory usage reporting, and the per-file processing step. Every new implementation must provide all of them, even those unrelated to retrieval, which makes alternative implementations expensive to write. A better division keeps the interface that rankers depend on small, as in the sketch above, and moves reporting and persistence into separate interfaces or helper functions.

Abstractions should follow implementations. The abstract classes were most useful where at least two real implementations existed, as with the standard and parallel indexes. Where only one implementation was ever written, the abstraction mostly added code. A reasonable rule is to write the first version concretely and extract an interface when the second version arrives, at which point it is clear which operations actually vary.

A second example: this site's search engine

The search box on this site is a small engine of the same kind, and it makes a useful comparison with the sketch, because it divides the work differently in one important place. It indexes the wiki's articles when the site starts, ranks them with BM25F over four fields (title, summary, headings, and body), answers Boolean and phrase queries, suggests spelling corrections, and gives instant results in the browser as a query is typed. The modules, as of November 2026, and the checks that hold them together:

Several of the article's points show up in this code.

The parser does not analyze. query.py turns (stemming OR lemmatization) AND NOT porter into a tree of operators over the words as typed. It never lowercases, removes stop words, or stems; the index does that when it evaluates the tree, with the same folding, stop list, and stemmer it used when indexing. The parser can therefore be tested on its own, as pure syntax, and it cannot analyze a word differently from the index, because it does not analyze at all. The remaining risk is the one this article warns of: inside SearchIndex, indexing and query evaluation call the shared folding, stop-list, and stemming functions from two separate code paths, so a divergence between them would surface only as a failing test, such as an article that no longer ranks first for its own title.

Ranking is not a separate component. Here the site departs from the sketch. BM25F is a method of SearchIndex, with its field weights and parameters as module constants, rather than a Ranker behind an interface. Swapping in another model would mean editing the index class. That is the lesson about abstractions from the course code applied in reverse: the site has one ranking function, tuned once, and an interface with a single implementation would add code without adding a choice. If a second ranker were ever wanted, for comparison on the known-item queries, extracting the interface would be the first step.

The export is a contract, and a test enforces it. BM25F gives each pair of a term and an article a fixed contribution to the score, so export() precomputes those weights, and the browser's code only adds them up. The browser therefore needs no stemmer and no BM25F of its own, and it cannot rank differently from the server unless the export or the adding is wrong. test_parity.py checks exactly that: it runs search-core.js under Node and SearchIndex.search in Python on every article title, cut off at several lengths, and on the known-item queries, and fails if any ranking differs. It is the same kind of test recommended above for two index implementations, applied across two languages.

The evaluator is a script over judged queries. evaluate.py measures mean reciprocal rank and success at 1 and 3 on a file of known-item queries, each paired with the one article that should come first. The BM25F weights were tuned on the first half of the file and checked on the held-out second half, the controlled comparison described at the start of this article, in miniature.

How production systems divide the work

The same divisions appear in established search libraries, which is good evidence that they reflect real boundaries in the problem. Apache Lucene, the library beneath Elasticsearch and Solr, separates an Analyzer (a tokenizer followed by a chain of token filters) from the index writer and from the scoring model, which is a pluggable Similarity class. Switching between classic TF-IDF scoring and BM25 means choosing a different Similarity, without rebuilding the index, provided the index stores the statistics the new model needs. Research toolkits built on Lucene, such as Anserini (Yang, Fang, and Lin, 2017), exploit this separation to make retrieval experiments reproducible: the same index can be searched with many ranking models, and every result can be regenerated from a documented configuration.

The design in this article is a small version of the same idea. It divides the system along the pipeline described in the architecture article, connects the parts through narrow interfaces, and keeps one analyzer for both documents and queries. A system built this way can grow through the whole course: each later topic adds or replaces one component, and the evaluator shows whether each change helped.

References

• Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, chapters 1–2 and 6. Cambridge University Press.
• Porter, M. F. (1980). An algorithm for suffix stripping. Program, 14(3), 130–137.
• Yang, P., Fang, H., and Lin, J. (2017). Anserini: Enabling the use of Lucene for information retrieval research. In Proceedings of SIGIR 2017, 1253–1256.
• Apache Lucene documentation: org.apache.lucene.analysis.Analyzer and org.apache.lucene.search.similarities.Similarity. https://lucene.apache.org/