How the Content Analysis Tools Work

Suppose a reviewer is about to revise The Porter Stemmer. Before changing a sentence, the reviewer wants to know:

• how long the article is and how its sentences read;
• which pages link to it;
• whether its references agree with other pages that cite the same papers;
• whether it breaks any of the style guide's rules;
• whether another page already explains the same ideas.

With about 470 pages on the site, answering those questions by reading would take days. How can a program answer them instead?

The answer is a set of command-line tools in tools/analysis/. They read every page, compute those properties, and write reports: JSON for scripts and agents, and short Markdown summaries for people. They run locally, are not deployed, and are not part of the build. Several of them reuse the text processing that the site search applies to the same pages, and two of them are retrieval methods turned on the site itself.

First comes what the tools share: one model of the site and the search engine's analysis of words. Then each tool in turn, with what it reports for the Porter article. Finally, what the tools found across the whole site, what has been done with it, and where they stop.

Every number about the site on this page comes from python -m tools.analysis all, run on November 12, 2026. A run takes about a minute. The comparison of stems and stop lists comes from tools/experiments/search/how-this-search-works.py.

One model of the site

Every tool reads the same model, built by tools/analysis/corpus.py. It loads app/wiki.py, the module the server uses to read pages, directly from its file. It therefore sees exactly the frontmatter, rendered HTML and links that a reader sees, without starting the web application. Loading all pages takes about 12 seconds.

For each page the model records:

• the frontmatter: slug, section, title, summary, tags, type, topic and, for reading notes, date, authors and source;
• the sections, each with its heading, level, anchor and word count;
• paragraphs, list items, captions and table cells as plain text;
• links, math, figures and other visuals (with whether each file exists), code blocks and tables;
• the raw entries of the References section.

Links come from the rendered HTML, so a wiki link written inside code or math is not counted, just as the server does not render it as a link. Word counts leave out code blocks, math, headings and the References section; inline code counts as words.

Words, the way search reads them

Where a tool compares words, it uses the search's analysis. tools/analysis/textnorm.py loads the site's Porter stemmer from app/porter.py. It tokenizes with the same pattern: accents removed, the text lowercased, then runs of the letters a to z and digits 0 to 9. When it compares content, it also drops numbers and single letters, which search indexes. The experiment script confirms that both give identical stems on every surface word in the search index, more than 23,000 of them. A tool that groups stemmed, stemming and stems together therefore groups them exactly as a search for any one of them would.

The stop list differs on purpose. Search keeps a short list of 33 words, because a word removed from the index can never be searched for. The analysis tools use 163 words, a superset that also drops words such as each, which and between. They compare pages by what they are about, and function words only add noise to that comparison. The sentence "Inverted indexes store the postings of each term." gives the search the terms invert, index, store, post, each, term, and the analysis tools the same without each.

Counting: inventory

The inventory writes one row per page, with counts of words, sections, visuals, code, math, tables, references and links, and sentence statistics. For the Porter article the row reads, in part:

The summary report then lists outliers, which is where a revision pass starts:

• course articles outside the style guide's range of 2,000 to 6,000 words;
• long articles with no visuals, and pages with the most words per visual;
• pages with the highest share of long sentences;
• code-heavy pages, and pages with no references.

None of these lists includes the Porter article.

Following links: links

The links tool builds the graph of page-to-page links. The entry points are the topic pages and the pages that the site's templates link by slug. From them the tool reports orphans (pages that nothing links to and that are not entry points) and pages that cannot be reached by following links. It also reports dead ends with no outgoing links, and planned links to pages not yet written. It checks each topic page against the articles that name it in their topic: field. It lists one-way links between articles of the same topic and counts links between courses and the library.

In November 2026 the graph had 4,538 links among 471 pages and 56 entry points. It had no orphans, no unreachable pages and no dead ends, and 15 planned links to 3 unwritten pages. The Porter article is reached from 13 pages and links 5.

Reading the references: refs

The refs tool parses every entry in every References section into authors, year, title, venue, volume, issue, pages, DOI and links, and clusters the entries into cited works. Entries join a cluster when they share a DOI or arXiv identifier, or the same first author, year and main title. Each work keeps every way it has been written and the pages that cite it.

Porter's 1980 paper is one work cited by six pages, written identically on all six. Elsewhere the clustering does real work. In November 2026 the tool found 2,469 entries naming 1,462 distinct works. It listed 43 pairs of works that may be the same, for a person to judge: the same first author with a similar title and a different year, or the same title under a different first author. The tool also matches in-text citations such as "(Porter, 1980)" to the same page's entries and reports citations with no entry and entries never cited. It is a heuristic, so its report is read rather than enforced. The clustered works are the data for a site-wide bibliography that is planned but not yet written.

Checking conventions: lint

The lint tool checks what a program can check. Findings come in three severities:

• error, for something broken: a missing figure file, a shortcode that did not expand, a missing summary, a topic: that names no topic page;
• warn, for a breach of a site convention: an em-dash, a word the style guide bans, a heading that skips a level, a missing References section, or two spellings of one term;
• info, for something worth a look: a dollar sign written as currency, a nested list, raw HTML.

Prose checks run with code and math masked and skip the References section, where cited titles keep their published wording. The spelling groups are in tools/analysis/lint_terms.json, and the report counts every form across the site, which is the evidence for choosing a house spelling. In November 2026 lint reported 62 findings. The 7 warnings were six pages using a second spelling of a term and one page without a References section. Of the 55 for information, 45 were dollar signs that stand for money.

The same explanation twice: overlap

The style guide asks that each concept be explained fully once and linked elsewhere. The overlap tool finds the places where that has not happened yet, and it does so with two techniques from the information retrieval course.

Its documents are paragraphs (at least 30 words, with consecutive list items of one section merged) and sections (each second- or third-level heading with its subsections, at least 80 words). It leaves out References, navigation sections, captions, table cells and code. Then it compares units in two ways.

• Similarity of vocabulary. Each unit becomes a TF-IDF vector over stemmed content words, with stop words, numbers and mathematics removed, and pairs are compared by cosine similarity. A paragraph pair is reported at a cosine of 0.55 or more, a section pair at 0.45 or more.
• Shared wording. Each paragraph also becomes its set of word 5-shingles, every run of five consecutive words, the technique of near-duplicate detection (Broder, 1997). Two measures compare the sets: the Jaccard coefficient, shared shingles over all shingles, and containment, shared shingles over the smaller set. A pair is a near-duplicate at a Jaccard coefficient of 0.35 or more, or a containment of 0.5 or more with at least 12 shingles in common. The sets are compared exactly through an inverted index rather than estimated with MinHash; at a few hundred pages the exact computation is cheap.

Each pair of pages then falls into one of four categories:

• near-duplicates, a paragraph repeated nearly word for word;
• parallel explanations, the same idea explained on both pages, which is the review list;
• expected overlap, such as a topic page summarizing its own article, the analogous theorems for groups and for rings in abstract algebra, or two pages describing the same dataset;
• weak matches, resting on a few shared words.

The Porter article has three partners: a weak match and two classed as parallel, one of them Stemming and Lemmatization. That match is between the two introductions, at a cosine of 0.50, and its strongest shared words are warm, warmer, stemmer, porter and lemmatizer. Each introduction opens on the same Arctic sentences from the architecture article and names the other page. This is a borderline case rather than an idea explained twice, and the two pages already link each other. A reviewer reading the pair would leave both as they are. Such calls are common. In a manual reading of 40 randomly sampled parallel pairs, about 26 were the same idea explained twice, 8 were borderline and 6 were not overlap at all.

Check your understanding. A paragraph of 40 words is copied into the middle of a 200-word paragraph on another page. Which of the two shingle measures flags the copy, and why does the other one miss it?

Containment flags it. The short paragraph has 36 shingles, and nearly all of them are shared, so containment, which divides by the smaller set, is close to 1. The Jaccard coefficient divides by the union, about 196 shingles from the longer paragraph, so it stays near 0.18, below the threshold. Of the 3 near-duplicates found in November 2026, two were caught by containment alone. One is a pair between the binary independence model and the Boolean model, with a Jaccard coefficient of 0.20 and a containment of 0.52.

Finding the terms: terms

The terms tool looks for definitions rather than similarity. Its candidates come from four places:

• bold terms in defining sentences ("called X", "X is ...");
• page titles;
• headings that several pages share;
• phrases of two to four words that recur on at least five pages.

Each candidate is normalized for case, plurals, hyphens and the spelling groups of lint, so that two spellings of one word count as one term.

For each term the tool chooses a canonical page. It prefers a page titled or tagged with the term, then the strongest definition, then the page that comes earliest in course order. It then lists the pages that use the term without linking any page that defines it, as candidate links. A term with a defining sentence that is used or defined on at least two pages becomes a glossary candidate. A single word needs stronger evidence, such as a page titled with it.

The word stem shows how the choice is made. Both the Porter article and Stemming and Lemmatization bold it, and neither is titled with it. The tool therefore chooses Stemming and Lemmatization, the earlier page in the course's reading order. In November 2026 the tool found 9,015 terms, 792 of them glossary candidates, and 750 candidate links. Its draft glossary, one candidate per line with a definition and a source page, is the raw material for a glossary page that is planned but not yet written.

Recording the review

The review itself is tracked in docs/review-ledger.tsv, a plain table with one row per page per review pass. Each row holds the date, the page, the pass, the reviewer, the checks made, a verdict and notes. Rows are written with python -m tools.analysis ledger add, and ledger coverage reports which pages a pass has reached. Git keeps the table's history. On November 12, 2026, a prose pass had reached 242 of the 471 pages, and a consolidation pass 10.

What the tools found

The run of November 12, 2026 gave:

The reports have already driven one consistency pass over the pages. They feed the site-wide review, now under way, and are meant to feed a glossary and a bibliography, both still planned.

Limits

• Every judgment is a heuristic. Overlap mistakes shared vocabulary for shared explanation, as the Porter pair shows, and terms sometimes picks a sentence that uses a bold word without defining it. The reports are lists for a person or an agent to read, not rules to enforce.
• The thresholds are settings at the top of tools/analysis/overlap.py. When a new dataset or series of pages shows up as a false parallel, it is added to tools/analysis/overlap_config.json rather than handled by raising a threshold.
• The tools see only what is on the pages. They cannot tell whether a number is right or a citation supports its sentence; that is the work of each article's experiment scripts and of the citation audits.
• Like the search, the analysis is for English text.

What we have learned

The tools turn questions a reviewer would ask about one page into measurements over all of them. They share one model of the corpus and the same tokenizer and stemmer as the site search. Two of them are retrieval methods turned inward: TF-IDF and cosine similarity find parallel explanations, and shingles find copied wording. The search engine that shares their analysis of words, from the index to the ranking, is described in How This Search Works.

References

• Broder, A. Z. (1997). On the resemblance and containment of documents. In Proceedings of Compression and Complexity of SEQUENCES 1997, 21–29. IEEE.
• Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, chapters 6 (TF-IDF weighting and cosine similarity) and 19 (near-duplicates and shingling). Cambridge University Press.
• Porter, M. F. (1980). An algorithm for suffix stripping. Program, 14(3), 130–137.

Code for this page: python -m tools.analysis all writes every report quoted here to tools/analysis/out/. The script tools/experiments/search/how-this-search-works.py compares the tools' stems and stop list with the search's.