{"repo":"spider-rs/spider-py","free":true,"listed":false,"github":"https://github.com/spider-rs/spider-py","clone":"git clone https://github.com/spider-rs/spider-py.git","description":"Spider ported to Python","language":"Rust","stars":109,"topics":["python","scraper","spider","web-crawler","headless-chrome","crawler"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"spider-py The spider project ported to Python. Getting Started 1. pip install spider rs View the examples to learn more. Development Install maturin pipx install maturin and python. 1. maturin develop Benchmarks View the benchmarks to see a breakdown between libs and platforms. Test url: https://espn.com libraries pages speed :--------------------------- :-------- :------ spider(rust): crawl 150,387 1m spider(nodejs): crawl 150,387 153s spider(python): crawl 150,387 186s scrapy(python): crawl 49,598 1h crawlee(nodejs): crawl 18,779 30m The benches above were ran on a mac m1, spider on linux arm machines performs about 2-10x faster. Issues Please submit a Github issue for any issues found.","default_branch":null,"files":null,"tree":[],"storefront":"/r/spider-rs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/spider-rs/spider-py/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}