{"repo":"jamesturk/scrapelib","free":true,"listed":false,"github":"https://github.com/jamesturk/scrapelib","clone":"git clone https://github.com/jamesturk/scrapelib.git","description":"⛏ a library for scraping unreliable pages","language":"Python","stars":212,"topics":["python","scraper","http"],"license":"BSD-2-Clause","category":"scrapers-browser-automation","readme_excerpt":"scrapelib is a library for making requests to less-than-reliable websites. This repository has moved to Codeberg, GitHub will remain as a read-only mirror. Source: https://codeberg.org/jpt/scrapelib Documentation: https://jamesturk.github.io/scrapelib/ Issues: https://codeberg.org/jpt/scrapelib/issues Features scrapelib originated as part of the Open States project to scrape the websites of all 50 state legislatures and as a result was therefore designed with features desirable when dealing with sites that have intermittent errors or require rate-limiting. Advantages of using scrapelib over using requests as-is: - HTTP(S) and FTP requests via an identical API - support for simple caching with pluggable cache backends - highly-configurable request throtting - configurable retries for non-permanent site failures - All of the power of the suberb requests library. Installation scrapelib is on PyPI, and can be installed via any standard package management tool. Example Usage","default_branch":null,"files":null,"tree":[],"storefront":"/r/jamesturk","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jamesturk/scrapelib/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}