Web crawler at scale

1 · Clarifying questions

Five minutes. A crawler can mean a search-engine index, a price scraper, a backup service, or a security scanner — the scope question matters more here than in most designs.

2 · Capacity math, on a napkin

Before any boxes. The numbers drive the frontier size, the bandwidth budget, and the storage tier choice.

The numbers tell the story before the architecture does. 1 GB/sec of sustained ingest is "this is a bandwidth problem before it is a CPU problem". 12 GB of Bloom filter is "we can keep dedup in memory; we don't need a database round-trip on every URL".

3 · API and data model

A crawler is mostly an internal system; the public surface is the seed-injection API and the operator dashboard. The data model is where the design choices live.

Endpoints (internal)

POST /v1/seeds                           # inject URLs into the frontier
{"urls": ["https://example.com/", ...], "priority": "normal"}
→ 202 {"accepted": 1024, "duplicates": 17}

GET  /v1/url/:hash                       # look up crawl status
→ 200 {"url":"...", "last_crawled":"...", "status":200, "etag":"..."}

POST /v1/policy/host                     # override politeness
{"host":"slow-site.example", "rps": 0.2}
→ 200

GET  /v1/stats                           # operator dashboard feed
→ 200 {"fetched_24h": 1.02e9, "errors_24h": 4.3e7, ...}

Schema

urls                                       -- canonical record per URL
  url_hash       BYTEA PRIMARY KEY          -- SHA-256(canonical_url)
  canonical_url  TEXT NOT NULL
  host           TEXT NOT NULL
  last_crawled   TIMESTAMP NULL
  etag           TEXT NULL
  content_hash   BYTEA NULL                 -- dedup near-duplicates
  status         SMALLINT NULL              -- last HTTP status
  priority       SMALLINT NOT NULL
  next_crawl_at  TIMESTAMP NULL
  INDEX (host, next_crawl_at)               -- scheduler pull

frontier                                    -- the live queue
  host           TEXT NOT NULL
  scheduled_at   TIMESTAMP NOT NULL
  url_hash       BYTEA NOT NULL
  priority       SMALLINT NOT NULL
  PRIMARY KEY (host, scheduled_at, url_hash)

host_politeness                             -- per-host gate
  host             TEXT PRIMARY KEY
  last_request_at  TIMESTAMP NOT NULL
  allowed_after    TIMESTAMP NOT NULL
  rps              REAL NOT NULL            -- crawl-delay-derived
  robots_fetched   TIMESTAMP NULL

crawled_content                             -- pointer to object store
  url_hash       BYTEA PRIMARY KEY
  fetched_at     TIMESTAMP NOT NULL
  content_addr   TEXT NOT NULL              -- s3://bucket/aa/bb/
  bytes          INT NOT NULL
  content_type   TEXT NOT NULL

The hot path reads host_politeness and writes to frontier. Raw bytes go to object storage keyed by content hash, which gives near-duplicate dedup for free.

4 · High-level architecture

The boxes-and-arrows. Crawlers split cleanly into the fetch loop (URL frontier → DNS → fetcher → store) and the discover loop (parser → dedup → frontier). The scheduler bridges them.

Two loops, one queue. The fetch loop drains the frontier through the politeness gate; the discover loop pushes newly-extracted URLs back into the frontier after dedup. The renderer is an opt-in path that runs about 10× slower than plain HTTP.

5 · The hard part — the URL frontier

The frontier is the queue of URLs waiting to be crawled. Sounds simple. The reason it is the heart of every serious crawler is that it has to satisfy three constraints at once: enforce politeness per host, schedule by priority, and answer "have I seen this URL?" in O(1).

Per-host queues

Politeness is per-host, so the frontier is too. Conceptually, one FIFO per host, with a wall-clock allowed_after on each queue. A single global queue would let one popular host starve the others; a single global rate cap would let one fast host hog the bandwidth and leave slow hosts unvisited.

In practice the frontier is implemented as a two-tier structure (Mercator-style): a back queue per host holds the URLs in order, and a front queue holds the host identifiers in priority order. The scheduler pops a host from the front queue, pops a URL from that host's back queue, and re-inserts the host into the front queue with the next allowed_after time. Heap-based, O(log H) where H is the number of hosts.

Dedup with Bloom filters

Before adding a URL to the frontier, ask: have we seen this URL before? At 10B URLs, a hash set in memory is too big (~600 GB), and a database round-trip per URL kills throughput. A Bloom filter at 10 bits per entry is ~12 GB total, fits in memory on one shard, and gives a ~1% false-positive rate. A false positive means we skip a URL we have not seen — annoying but not fatal; the URL gets re-discovered later through another link.

Canonical URLs

Same page, many URLs: ?utm_source=…, trailing slashes, scheme variants, session IDs. Canonicalise before hashing. Strip known tracking parameters, normalise the host case, sort query parameters, follow <link rel="canonical"> when the parser finds one. Without canonicalisation, dedup misses by 2–5×.

Priority and freshness

Front-queue priority is a combination of static importance (PageRank-like) and freshness deficit (time since last crawl ÷ expected re-crawl interval). News fronts float to the top; long-tail product pages sink. Re-crawl intervals are learned per page from observed change rate — pages that change every fetch get re-fetched frequently; pages that have not changed in a year get a 30-day TTL.

6 · Failure modes & runbook

The open web is hostile. Most of a crawler's code is failure handling, not happy path. The interview-room version of the operational story.

7 · Cost & operability

A ballpark in 2026 cloud prices. The cost shape is unusual: bandwidth and storage dominate; CPU is comparatively cheap.

8 · Trade-offs & "what would you change at 10×"

The interviewer's last 5 minutes. These are the answers that distinguish a competent design from a thoughtful one.

Further reading

• Heydon & Najork — "Mercator: A Scalable, Extensible Web Crawler" (1999). The canonical paper on the two-tier frontier. Old but the model is still right.
• Olston & Najork — "Web Crawling" (2010). A survey covering politeness, freshness, dedup, and the trade-offs between them. The best single reference if you read only one.
• Common Crawl — monthly crawl reports. Public data and operator writeups from one of the few open large-scale crawlers; useful for sanity-checking your numbers.
• Google — "Sitemaps protocol". The other half of the freshness story; respecting sitemaps cuts wasted fetches dramatically.
• Cloudflare — "How we built our internet-scale crawler" (2023). A modern operator's view; covers bot detection and the politeness-versus-freshness tension.
• Adjacent: Bloom filter simulator. The memory–FPR curve you cite when defending the dedup choice.
• Adjacent: How DNS works. A crawler is the worst-case DNS client; caching is non-optional.
• Adjacent: Rate limiter simulator. The per-host politeness gate is the same algorithm at a smaller scale.