Monitoring Websites

A search engine's crawler tries to fetch as much of the web as it can. Most crawling programs written by individuals do the opposite. They watch a handful of pages, a price, a course listing, a status page, a government notice, and report when something changes. The only crawler I had written before this course was one of these, in PHP, years ago, with everything written by hand. This article applies the ideas of the topic to that small problem. Nearly all of them carry over, scaled down: politeness, conditional requests and revisit rates, and fingerprints of the content that matters.

Check for a better source first

Before writing code that reads pages designed for people, look for something designed for programs:

• Feeds. Many sites that publish updates, such as news, blogs, release notes, and announcements, offer an RSS or Atom feed, often linked in the page's HTML header. A feed lists recent changes directly.
• Interfaces. Many sites offer an API, a documented interface that returns data, often in JSON. Scripted pages usually load their data from such an interface even when it is not documented, and the browser's developer tools show the requests.
• Sitemaps. A sitemap, often named in robots.txt, lists a site's pages, often with last-modified dates.
• Notifications. Some services will send email or call a web address supplied by the subscriber when something changes.

These are more stable than page layouts, cheaper for the site, and usually permitted explicitly. Also read the site's terms of service. Some forbid automated access, and robots.txt is not the only statement of a site's wishes.

The design

A monitor for one page needs only a few steps, run on a schedule:

1. Check robots.txt (Koster, Illyes, Zeller, and Sassman, 2022), and stop if the page is disallowed.
2. Fetch the page conditionally (Fielding, Nottingham, and Reschke, 2022, §13), sending the ETag and Last-Modified values saved from the last fetch. If the server answers 304 (Not Modified), stop: nothing has changed and almost nothing was transferred.
3. Extract the part that matters, with a CSS selector, and normalize its whitespace.
4. Fingerprint it with a hash, and compare with the saved fingerprint.
5. Save the new state, including the validators and a copy of the text, and report any change.

Extracting a part of the page before fingerprinting is the key decision. Whole pages change constantly for reasons that do not matter: timestamps, advertisements, rotating recommendations, hidden security tokens in forms. A fingerprint of the whole page would report a change on nearly every visit. A fingerprint of the one element being watched reports only changes to it.

A tested implementation

The following is the core of tools/experiments/crawling/monitor.py in the site repository, using the requests package and Beautiful Soup (beautifulsoup4; Richardson).

AGENT = "omnipedia-monitor/0.1 (+https://omnipedia.tech)"

def check(url, selector=None, state_file="monitor_state.json"):
    path = Path(state_file)
    state = json.loads(path.read_text()) if path.exists() else {}
    old = state.get(url, {})
    if not allowed(url):                      # robots.txt check
        return "disallowed by robots.txt"

    headers = {"User-Agent": AGENT}
    if "etag" in old: headers["If-None-Match"] = old["etag"]
    if "last_modified" in old: headers["If-Modified-Since"] = old["last_modified"]
    r = requests.get(url, headers=headers, timeout=20)
    if r.status_code == 304:
        return "not modified (304)"
    r.raise_for_status()

    soup = BeautifulSoup(r.text, "html.parser")
    part = soup.select_one(selector) if selector else soup.body or soup
    if part is None:
        return f"selector {selector!r} matched nothing; the page layout may have changed"
    text = " ".join(part.get_text(" ").split())
    digest = hashlib.sha256(text.encode()).hexdigest()
    # ... save digest, text, and the new validators, then compare with old["digest"]

It was tested against a page served from a local test server, watching an element with the selector #price:

Run 3 shows why the selector matters: the page had changed, the server sent it in full, and the monitor correctly ignored the change. Run 5 covers the most common way monitors fail. When a site is redesigned, the selector stops matching, and a monitor that treated "nothing found" as empty text would either report a false change or, worse, silently watch nothing.

How often to check

The freshness model gives a monitor's schedule a simple arithmetic. Suppose the watched element changes on average once a week, at random times, as a Poisson process, and the monitor checks it every $I$ hours. Two quantities matter: how long a change goes unnoticed, and how many requests the monitor makes.

The first has an exact answer that does not depend on the change rate. Changes of a Poisson process fall uniformly in time, so a change is equally likely to happen at any point between two checks, and the time until the next check is uniform between 0 and $I$. The mean delay before a change is noticed is $I/2$. The second is just $8{,}760 / I$ checks a year, and the fraction of them that find anything is the probability of at least one change in $I$ hours, $1 - e^{-\lambda I}$ with $\lambda = 1/168$ per hour.

Going from daily to five-minute checks multiplies the requests by 288 to cut the delay from 12 hours to 2.5 minutes, and 99.95 percent of those checks find nothing. Whether that is worth it depends on what a delay costs, a concert ticket going on sale against a policy document being revised, but the table makes the price visible. Conditional requests lower the price of the empty checks, since a 304 response carries no body, though each still costs the server a request. A simulation of one year of weekly changes, 30 runs, agreed with the $I/2$ delays: the median of the runs' mean delays was 29.4 minutes for hourly checks and 181 minutes for checks every six hours. The table is computed by tools/experiments/crawling/monitor_schedule.py.

Practical points

• Schedule rather than loop. Run the script from cron on Linux or macOS, or Task Scheduler on Windows, rather than leaving a program running with a sleep loop. A failed run then does not stop future ones.
• Choose the interval from the change rate. A page that changes weekly does not need checking every minute. The freshness analysis applies: frequent checks of a slowly changing page buy almost nothing. Hourly or daily is right for most purposes.
• Identify the monitor. Use a User-Agent naming the program, with a way to contact its operator, so the site's operator can see what is visiting.
• Back off on errors. On status 429 or 503, or repeated failures, wait longer before the next attempt. Never retry in a tight loop.
• Know the robots.txt parser's limits. Before Python 3.13.14 and 3.14.5, both released in 2026, urllib.robotparser applied the first matching rule rather than the most specific one, as described in politeness and robots.txt. It also ignored wildcards in paths, so it could be either stricter or more permissive than the standard. Older versions remain in wide use, so checking the version, or reading the target's robots.txt by eye, is a sensible check.
• Scripted pages. If the watched content does not appear in the HTML the server sends, the page builds it with JavaScript. Look for the underlying data request first. Rendering with a headless browser, through a library such as Playwright, works but is far heavier, for the monitor and the site alike, as described in the deep web.
• Keep history if it matters. A JSON file of last states is enough for a few pages. A small SQLite table of every observed version, with timestamps, makes it possible to see when and how often something changed, and difflib in the Python standard library can show exactly what changed between versions.

References

• Fielding, R., Nottingham, M., and Reschke, J. (2022). HTTP Semantics. RFC 9110, Internet Engineering Task Force. Sections 8.8 and 13 define validators and conditional requests.
• Koster, M., Illyes, G., Zeller, H., and Sassman, L. (2022). Robots Exclusion Protocol. RFC 9309, Internet Engineering Task Force.
• Richardson, L. Beautiful Soup documentation. Published on the web at crummy.com.