The Deep Web

A crawler finds pages by following links, so it can only reach pages that something links to, and only in the form a plain HTTP request returns. Everything else is the deep web, also called the hidden or invisible web: content that is publicly reachable, or reachable by some users, but not by following links from the pages a search engine already knows. The deep web is not the same as the "dark web", which refers to sites reachable only through anonymizing networks such as Tor. Most of the deep web is ordinary: library catalogs, flight schedules, court records, product databases, and anything behind a login.

What a crawler cannot reach

The course divided deep web content into three kinds.

Private sites

Some content is meant to be private. Email, bank accounts, company intranets, and course management systems require a login, and no public page links into them. A crawler should not try to reach this content, and a general web crawler has no credentials with which to do so. Search over private content is provided instead by the systems that hold it, as in the enterprise and personal search applications.

Form results

A great deal of public information is reached only by filling in a form. A flight search, a library catalog, a patent database, or a government registry generates its result pages from a database in response to the values a user enters. There may be no link to any individual result, and the number of possible queries is effectively unlimited. A crawler that only follows links sees the form and nothing behind it.

Scripted pages

Many sites send a nearly empty HTML page and build the visible content in the browser with JavaScript, fetching data from the server as the page runs. A crawler that downloads the HTML and parses it, as the crawl loop describes, sees only the empty shell and the script tags. Links created by scripts are equally invisible.

A flight-status page built this way might send only this much HTML:

<div id="flights">Loading...</div>
<script src="/static/status.js"></script>

In a browser, status.js runs, requests /api/departures?airport=RDU, receives a list of flights as JSON, and writes a table of them into the flights element. A crawler that only parses HTML indexes a page whose entire text is "Loading...", and finds no links to the individual flights. The JSON request, by contrast, has a URL of its own and returns the data in a form a program can read directly, which is why it is often the better target. Flash, a browser plug-in, caused the same problem until browsers stopped supporting it.

How large is it?

Bergman (2001) estimated that the deep web held 400 to 550 times as much content as the surface web indexed by search engines, about 7,500 terabytes against 19 terabytes, spread across some 200,000 sites. The estimate was extrapolated from a sample of large databases and counted raw data volume, so it says more about the size of the databases behind forms than about the number of useful documents. A later survey by He, Patel, Zhang, and Chang (2007) estimated about 307,000 deep web sites in 2004, offering some 450,000 databases. Whatever the exact figure, the conclusion is the same: link-following reaches only a small part of what is on the web.

Reaching some of it

Surfacing form results

A crawler can try to fill in forms itself. Raghavan and Garcia-Molina (2001) built HiWE, a crawler that analyzed a form's fields and filled them with values drawn from a task-specific list. Google's approach, described by Madhavan and colleagues (2008), was to surface form results: for each form, choose combinations of inputs that produce distinct, informative result pages, generate the URLs of those result pages, and add them to the ordinary crawl. A search box is filled with words taken from the site's own pages, and menus with their listed values. The result pages then become ordinary documents in the index, reachable by any query. The approach works only for forms submitted with GET, whose results have their own URLs, and only covers a sample of what each database holds.

Why forms cannot simply be enumerated

Consider a used-car search form with three menus, make (40 options), body style (8) and year (20), and a text box for a zip code. With the text box left empty, the menus alone give $40 \times 8 \times 20 = 6{,}400$ combinations. At a polite rate of one request every 10 seconds, submitting them all takes 17.8 hours, for one form on one site, and the text box multiplies that by every zip code. Many of the combinations return nothing, and many return the same cars sorted differently or split across pages.

Madhavan and colleagues (2008, §4) describe the choice in terms of query templates. A template names the inputs that will be varied, its binding inputs, and leaves the rest at their defaults. The template {make} generates 40 submissions, the template {make, year} generates 800, and the number of binding inputs is the template's dimension. Instead of submitting every combination, the crawler searches for templates whose results are worth indexing.

A template is judged by an informativeness test. The crawler submits the template's queries, computes a signature of each result page's content, and counts the distinct signatures. If the result pages are mostly distinct, the template retrieves different records for different inputs and is informative; if many pages have the same signature, it is not. Formally, with $G$ the set of submissions a template generates and $S$ the set of distinct signatures of their result pages, the template is informative if $\lvert S \rvert / \lvert G \rvert \ge \tau$, and Madhavan and colleagues used $\tau = 0.25$ (§4.1, §4.4). Suppose, for illustration, the following counts for the car form:

The template {style} is uninformative because, in this hypothetical form, the default make returns no records at all, so every body style gives the same "no results" page. This is the kind of form Madhavan and colleagues describe, with one primary input and others that only refine it. Their algorithm, ISIT, starts from templates of dimension 1 and extends only the informative ones by one more input at a time, so the uninformative {style} is never extended. A form with 6 inputs has $2^6 - 1 = 63$ possible templates; in their experiments ISIT tested 17.43 per such form on average and found 7.17 informative. They also limited templates to dimension 3 and to at most 10,000 generated URLs each (§4.2 to §4.3). The result pages of the informative templates, with duplicate signatures removed, are what enters the index.

Surfacing raises politeness questions. Every generated query costs the site a database search, and a form designed for people choosing one flight was not designed for a crawler choosing thousands. The rate limits of politeness apply with extra force.

Rendering scripts

To see a scripted page as a person does, a crawler must run it, by loading the page in a headless browser, a browser with no display, and reading the document after the scripts have run. Major search engines now render pages this way. Rendering costs far more than parsing HTML, in processing time and in the additional requests the scripts make, so crawlers typically fetch and parse first and render later, from a separate queue. Google, for example, queues every successfully fetched page for rendering in a headless Chromium, and a page may wait there for some time before its scripts are run. For a small crawler or a site monitor, the alternative is often better: the data a script displays usually comes from a request to the site's own interface, which can be fetched directly.

Sitemaps and feeds

The simplest route into the deep web is for the site to list its pages. The Sitemaps protocol, introduced by Google in 2005 and adopted jointly with other search engines in 2006 at sitemaps.org, lets a site publish an XML file of its URLs, with optional last-modified dates. A single sitemap file may list up to 50,000 URLs and be at most 50 MB uncompressed, and an index file can point to as many as 50,000 sitemaps. A minimal sitemap with one entry looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://cars.example/listing/48213</loc>
    <lastmod>2026-11-02</lastmod>
  </url>
</urlset>

Only <loc> is required. The optional <lastmod> is what lets a crawler skip pages that have not changed since its last visit, as described in freshness and recrawling. A product catalog that is reachable only through search can list every product page in its sitemap, and a crawler can reach them all without filling in a form. A sitemap is named in robots.txt or submitted to search engines directly.

The limit that remains

Even with all three techniques, a large part of the web stays out of reach, and much of it should. Private content is private by design. Databases with millions of records cannot be fully surfaced through a search form, and a crawler that tried would be indistinguishable from an attack. The practical lesson for anyone building a crawler is to check, before writing any scraping code, whether a site offers a sitemap, a feed, or an interface intended for programs. These are usually more complete, more stable, and more welcome than anything a crawler can extract from pages designed for people.

References

• Bergman, M. K. (2001). The deep web: Surfacing hidden value. Journal of Electronic Publishing, 7(1).
• He, B., Patel, M., Zhang, Z., and Chang, K. C.-C. (2007). Accessing the deep web. Communications of the ACM, 50(5), 94–101.
• Madhavan, J., Ko, D., Kot, Ł., Ganapathy, V., Rasmussen, A., and Halevy, A. (2008). Google's deep web crawl. Proceedings of the VLDB Endowment, 1(2), 1241–1252.
• Raghavan, S., and Garcia-Molina, H. (2001). Crawling the hidden web. In Proceedings of VLDB 2001, 129–138.
• Sitemaps.org. Sitemaps XML format. Protocol documentation, 2006 and later.