{"repo":"BitingSnakes/silkworm","free":true,"listed":false,"github":"https://github.com/BitingSnakes/silkworm","clone":"git clone https://github.com/BitingSnakes/silkworm.git","description":"Async web scraping framework on top of Rust. Works with Free-threaded Python (`PYTHON_GIL=0`).","language":"Python","stars":74,"topics":["rust","web-scraping","free-threaded-python","scraping"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"silkworm-rs Async-first web scraping framework built on wreq (HTTP with browser impersonation) and scraper-rs (fast HTML parsing). Silkworm gives you a minimal Spider/Request/Response model, middlewares, and pipelines so you can script quick scrapes or build larger crawlers without boilerplate. NEW : Use silkworm-mcp to build scrapers. Features - Async engine with configurable concurrency, priority-aware queueing, bounded backpressure (defaults to concurrency 10 ), and per-request timeouts. - wreq-powered HTTP client: browser impersonation, redirect following with loop detection, query merging, and proxy support via request.meta[\"proxy\"] . - Optional OnionLink client integration for scraping Tor v3 .onion sites without routing through wreq. - Optional Servo rendering via ServoFetchClient for JavaScript-rendered pages without changing the default HTTP client. - Typed spiders and callbacks that can return items or Request objects; HTMLResponse ships helper methods plus Response.follow to reuse callbacks. - HTML-to-Markdown conversion via fast-h2m , including rich full , lean minimal , and streaming modes. - Middlewares: User-Agent rotation/default, proxy rotation, cookie jars with save/load, retry with exponential backoff + optional sleep codes, flexible delays (fixed/random/custom), robots.txt delay enforcement, SkipNonHTMLMiddleware to drop non-HTML callbacks, and CloudflareCrawlMiddleware for Browser Rendering crawl jobs. - Pipelines: JSON Lines, SQLite, XML (nested data pre","default_branch":null,"files":null,"tree":[],"storefront":"/r/BitingSnakes","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/BitingSnakes/silkworm/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}