Designing bounded crawls for production

Define the boundary before the first request

A production crawl needs an explicit depth boundary and URL scope. Those controls turn an open-ended link graph into a job that callers can reason about and operators can safely schedule.

Start from the smallest useful seed set, choose the allowed scope, and set a sitemap limit when sitemap discovery belongs in the collection plan.

Keep scope explicit

Same-origin and same-domain rules solve different jobs. Pair the selected scope with include and exclude patterns so pagination, article routes, and unwanted account or search pages are handled deliberately.

• Use same origin when scheme and host changes are not expected.
• Use same domain when approved subdomains belong to the collection job.
• Exclude login, account, checkout, and internal search paths unless they are a verified target.

Treat the crawl as a job

Create the crawl once, keep its identifier, and read progress and results through that durable job resource. A web request should not need to remain open while many pages are collected.

curl -X POST "$SPIDER_BASE_URL/v1/crawl" \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/docs"],"max_depth":2,"scope":"same_hostname"}'

Retain page-level context

A final job count is useful, but each accepted page still needs its own URL, content, status, request evidence, and discovery context. Keep those records stable so downstream indexing and review do not depend on queue internals.