Browser API

Run browser-backed scrapes with public-safe sessions, profile pools, and profiles.

Quick start

Use browser scrape when a page needs rendering. Public browser controls are limited to wait, iframe, shadow DOM, and virtual scroll behavior.

curl -X POST "$SPIDER_API_BASE/v1/browser/scrape" \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Idempotency-Key: browser-example-01" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog",
    "browser": {
      "wait_for_selector": ".post-card",
      "process_iframes": true,
      "flatten_shadow_dom": true,
      "virtual_scroll": {
        "container_selector": ".feed",
        "scroll_count": 5,
        "scroll_by": "page_height"
      }
    },
    "extraction": {
      "type": "json_css",
      "schema": {
        "name": "blog_posts",
        "base_selector": ".post-card",
        "fields": [
          { "name": "title", "selector": "h2", "type": "text", "transform": "strip" },
          { "name": "path", "selector": "a", "type": "attribute", "attribute": "href" }
        ]
      },
      "computed_fields": [
        { "name": "url", "operation": "template", "template": "https://example.com{path}" }
      ]
    }
  }'

Authentication

Use a bearer token.

Create and manage product API keys in Product Console settings. The docs page itself is public.

Response shape

Rendered result

Page content and structured items returned after DICloak-backed browser execution completes.

Safe browser controls

browser.wait_for_selector, process_iframes, flatten_shadow_dom, and virtual_scroll map to human_behavior without exposing arbitrary JavaScript, cookies, storage state, proxy URLs, user agents, or DICloak profile IDs.

Structured extraction

Crawl4AI executes the same allowlisted schema used by Extract and Crawl over rendered HTML.

Profile pools

Current /v1/browser/profile-pools inventory without provider IDs or raw profile payloads.

Diagnostics

Browser scrape returns PublicPageResult.diagnostics; use result.diagnostics.adaptive_extraction for selector memory / adaptive extraction diagnostics, result.diagnostics.browser_debug for sanitized browser debug capture counts, result.diagnostics.browser_failure for sanitized browser failure category/stage diagnostics, and result.diagnostics.content_quality for content quality diagnostics.

Reference

Endpoint reference

OpenAPI-backed details for browser scrape, sessions, profile pools, and error responses.

OpenAPI endpoint details

Interactive schemas, request bodies, response objects, and public error contracts load here.