Crawl API

Create asynchronous crawls, check job status, and page through crawl results.

Quick start

Create a crawl with 1 to 100 HTTP(S) seed URLs without URL userinfo. Public Crawl requires a materialized plan before quota reservation; then poll status and list results by crawl ID. scope defaults to same_hostname. same_hostname follows only the exact hostname. same_domain follows hosts sharing the same registrable domain, including subdomains. same_origin additionally requires the same scheme and effective port. all accepts any valid HTTP(S) URL.

curl -X POST "$SPIDER_API_BASE/v1/crawl" \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Idempotency-Key: crawl-example-01" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/blog"],
    "max_depth": 1,
    "scope": "same_hostname",
    "browser": {
      "wait_for_selector": ".post-card",
      "process_iframes": true,
      "flatten_shadow_dom": true,
      "virtual_scroll": {
        "container_selector": ".feed",
        "scroll_count": 5
      }
    },
    "extraction": {
      "type": "json_css",
      "schema": {
        "name": "blog_posts",
        "base_selector": ".post-card",
        "fields": [
          { "name": "title", "selector": "h2", "type": "text", "transform": "strip" },
          { "name": "path", "selector": "a", "type": "attribute", "attribute": "href" }
        ]
      },
      "computed_fields": [
        { "name": "url", "operation": "template", "template": "https://example.com{path}" }
      ]
    }
  }'

Authentication

Use a bearer token.

Create and manage product API keys in Product Console settings. The docs page itself is public.

Response shape

Crawl job

Accepted job metadata with IDs used by status and results endpoints.

Safe browser controls

Public crawl may use the same browser wait, iframe, shadow DOM, and virtual_scroll behavior controls while remaining an asynchronous job API.

Status

Progress, terminal state, counters, and failure information.

Results

Paginated public page results created by the crawl.

Items

GET /v1/crawl/{crawl_id}/items returns extracted records with source page provenance.

Diagnostics

Use crawl.diagnostics.block_reasons for crawl blocked diagnostics, crawl.diagnostics.retry.categories for retry taxonomy, items[].diagnostics.adaptive_extraction for selector memory / adaptive extraction diagnostics, items[].diagnostics.browser_debug for sanitized browser debug capture counts, and items[].diagnostics.browser_failure for sanitized browser failure category/stage diagnostics.

Reference

Endpoint reference

OpenAPI-backed details for crawl creation, status polling, result pagination, and errors.

OpenAPI endpoint details

Interactive schemas, request bodies, response objects, and public error contracts load here.