SpiderFly journal

Web data field notes

Practical guides for fetching, rendering, crawling, and shaping public web data for production systems.

3 field notes

Newest first

When a web data request needs a browser

A practical way to choose between direct fetch and browser rendering without paying browser cost for every page.

Define scope, depth, and result ownership before a crawl starts so asynchronous collection stays predictable.

Separate acquisition from extraction, then validate a typed result before it becomes agent context.