Politeness and robots.txt

A crawler is a guest on every server it visits. Each request costs the server something, and the crawler gets the benefit. Politeness is the set of practices that keep that cost acceptable to the server's owner. It matters ethically, since a careless crawler can slow or break a small site, and practically, since site operators block crawlers that misbehave. Manning, Raghavan, and Schütze (2008, §20.1.1) list it with robustness as one of the two features a crawler must have.

Rate limits

The most important rule is the simplest: requests to one server must not be sent too quickly. Common practice is to keep at most one connection open to a host at a time and to wait between requests to the same host. A common heuristic, described by Manning, Raghavan, and Schütze (2008, §20.2.3) and used in production crawls by the Mercator crawler (Najork and Heydon, 2002), sets the wait adaptively, at about ten times the duration of the last fetch from that host. A server that responds slowly, perhaps because it is busy, is automatically visited less often.

The rule has a simple consequence. If a fetch from a host takes $d$ seconds and the crawler then waits $10d$ before the next, each cycle lasts $11d$, of which the server spends $d$ serving the crawler. Whatever the server's speed, the crawler occupies it at most $1/11$, about 9 percent, of the time. In absolute terms the rate adapts: a host that answers in 0.1 seconds can be fetched once every 1.1 seconds, about 78,500 times a day, while one that takes half a second is fetched at most once every 5.5 seconds, about 15,700 times a day. If the server slows down under load, the crawler slows down with it, without having to detect the load.

A crawler with many threads must enforce this across all of them. The URL frontier does it by keeping a separate queue for each host and giving each queue to one thread at a time. The timeline below comes from the small frontier traced in crawler architecture: one thread, four hosts with different response times, and the ten-times rule.

Two things are visible in it. Requests to any one host are spread out, while the thread moves between hosts to fill the waits. And with only four hosts, the thread is idle much of the time, busy for 2.4 of the 6.2 seconds, because every host is still in its waiting period. Politeness costs a crawler nothing in total throughput only when it has many hosts to choose from.

Servers can also ask a crawler to slow down. A response with status 429 (Too Many Requests) or 503 (Service Unavailable), especially with a Retry-After header, means the crawler should back off from that host, not retry at once.

Identifying the crawler

Every HTTP request carries a User-Agent header. A polite crawler sends a name that is distinct from any browser's, with a URL or email address where the site's operator can learn what the crawler is and how to contact its owner. Site operators who find unfamiliar traffic in their logs can then decide whether to allow it, write robots.txt rules for it, or ask for changes, rather than blocking it outright.

The Robots Exclusion Protocol

Martijn Koster proposed the Robots Exclusion Protocol in 1994, and crawlers followed it for nearly three decades as an informal convention. It was standardized as RFC 9309 in 2022 (Koster, Illyes, Zeller, and Sassman, 2022). A site publishes its rules in a plain text file at the root of the host, such as https://example.com/robots.txt. A crawler fetches that file before anything else on the host and obeys it.

The file consists of groups. Each group begins with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow rules giving path prefixes. A group for * applies to any crawler without a group of its own. A Sitemap line, which belongs to no group, gives the location of a sitemap listing the site's URLs.

User-agent: *
Disallow: /private/
Disallow: /confidential/
Disallow: /other/
Allow: /other/public/

User-agent: FavoredCrawler
Disallow:

Sitemap: http://mysite.com/sitemap.xml.gz

Under this file, crawlers in general may fetch anything except paths under /private/, /confidential/, and /other/, but they may fetch paths under /other/public/. A crawler named FavoredCrawler is matched by its own group, whose empty Disallow forbids nothing, so it may fetch everything.

Matching rules

RFC 9309 settles several points that the original convention left open:

• The most specific rule wins. When several rules match a path, the one with the longest path pattern applies, regardless of the order of the lines. That is why /other/public/c.html is allowed above: Allow: /other/public/ is longer than Disallow: /other/. If an Allow and a Disallow match equally, Allow applies.
• A crawler obeys one group. It uses the group whose User-agent matches its name, compared without regard to case, and only if there is none, the * group. The rules of its own group are not combined with those of the * group, though several groups naming the same crawler are merged into one.
• Paths are case-sensitive, and * (any characters) and $ (end of the path) may be used in patterns.

Applied to the example file, a crawler without a group of its own uses the * group, finds every rule whose path is a prefix of the requested path, and keeps the longest:

The last row is a reminder that rules are prefixes of the path as characters, not directories. Disallow: /other/ does not match /otherwise.html, because of the final slash, but Disallow: /other would.

When the file is missing or unreachable

The status code of the robots.txt request matters:

• Success (2xx): parse and obey the file.
• Unavailable (4xx, such as 404): the site has no rules, and the crawler may fetch anything.
• Unreachable (5xx or network failure): the crawler must assume that everything is disallowed. If the error persists for a long period, for which the RFC gives 30 days as an example, it may fall back to a cached copy, or treat the file as unavailable.

The asymmetry is deliberate. A missing file is a statement that there are no restrictions. A server error says nothing, and an unreachable server is a poor one to start crawling. Crawlers may cache robots.txt, but generally for no more than 24 hours, so that changes take effect within a day.

Crawl-delay and other extensions

Many robots.txt files contain a Crawl-delay line giving a number of seconds to wait between requests. It is not part of RFC 9309. Some crawlers honor it and others, including Google's, ignore it. A crawler that honors it should still apply its own limits as well.

What robots.txt is not

The protocol is a request, not a lock. A malicious crawler can ignore it, and a robots.txt file listing private paths tells everyone where they are. It is not a security mechanism, and content that must not be fetched needs authentication. It is also not a statement of legal permission. Terms of service, copyright, and data protection law apply whether or not a path is disallowed.

Pitfalls in Python's parser

Python's standard library includes a robots.txt parser, urllib.robotparser, which predates RFC 9309. Until 2026 it differed from the standard in several ways:

• It applied the first matching rule in the file, not the most specific one. Run on the file above, it disallowed /other/public/c.html, because the Disallow: /other/ line comes first. Under RFC 9309, the longer Allow rule wins and the path is allowed. In the opposite order, Allow: /a/ followed by Disallow: /a/b/, it allowed /a/b/x, which the standard disallows.
• It did not support the wildcards * and $ in paths, treating them as ordinary characters, so a rule such as Disallow: /*.pdf matched nothing and every PDF was allowed.
• It did not merge groups that name the same crawler. Only the first such group was used.

Python 3.15 and the maintenance releases 3.13.14 and 3.14.5 implement the longest match, the wildcards, and the merging of groups. Older versions, including every release of 3.12 and earlier, still behave as described, and their differences err in both directions, so the parser can fetch more than a site has asked to allow, as well as less. A crawler that must follow the standard on such a version needs a parser that implements it, such as the protego package, or at least a reading of the target site's robots.txt by eye to see whether any of these cases arise. In every version, a 401 or 403 response for robots.txt makes the parser disallow the whole site, where RFC 9309 treats any 4xx response as meaning that there are no rules. The checks are in tools/experiments/crawling/examples.py in the site repository.

References

• Koster, M. (1994). A standard for robot exclusion. Published on the web at robotstxt.org.
• Koster, M., Illyes, G., Zeller, H., and Sassman, L. (2022). Robots Exclusion Protocol. RFC 9309, Internet Engineering Task Force.
• Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, sections 20.1–20.2. Cambridge University Press.
• Najork, M., and Heydon, A. (2002). High-performance web crawling. In J. Abello, P. M. Pardalos, and M. G. C. Resende (Eds.), Handbook of Massive Data Sets, 25–45. Kluwer Academic Publishers.