{"repo":"spider-rs/spider","free":true,"listed":false,"github":"https://github.com/spider-rs/spider","clone":"git clone https://github.com/spider-rs/spider.git","description":"Get web data for AI agents and LLMs","language":"Rust","stars":2658,"topics":["crawler","rust","spider","headless-chrome","scraping","automation","ai-agent","web-crawler","web-scraping","web-data"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Spider The fastest web crawler and scraper for Rust. spider.cloud · Guides · Docs · Examples · Discord --- Spider is a concurrency-first crawling engine written in Rust. It streams pages as they arrive, renders JavaScript only on pages that need it, and scales from a single script to a distributed fleet without changing your code. The same engine runs Spider Cloud, so you can prototype locally and move to managed infrastructure with one config change. Start in the cloud The hardest part of crawling at scale isn't the code. It's the proxies, headless browsers, and constant anti-bot churn. Spider Cloud runs all of that for you behind the same API. Get a free API key → (no card required) Smart mode routes through proxies first and escalates to the unblocker only on pages that fight back, so you pay for bypass only where it's needed. Or run it locally No key, no service. Just the crawler. Pages stream in as they're fetched. The crawler finds the links, stays inside the limits you set, and stops on its own. How it works Spider runs HTTP-first and launches headless Chrome only when a page needs JavaScript. Both the HTTP and Chrome paths stream, so pages come back as they're fetched instead of batching at the end. The same API drives one async task or a distributed worker fleet, and the concurrency model doesn't change between them. Proxies, retries, rate limiting, and stealth are built in. Install You want… Run --- --- Rust library cargo add spider Command-line tool cargo install s","default_branch":null,"files":null,"tree":[],"storefront":"/r/spider-rs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/spider-rs/spider/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}