{"repo":"elliotgao2/gain","free":true,"listed":false,"github":"https://github.com/elliotgao2/gain","clone":"git clone https://github.com/elliotgao2/gain.git","description":"Web crawling framework based on asyncio.","language":"Python","stars":2019,"topics":["python","crawler","spider","asyncio","uvloop","aiohttp"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"gain Async web crawling framework for everyone. Built on asyncio , aiohttp , and lxml / pyquery . Declare items and parsers; gain handles the concurrency, retries, and persistence. Install Linux users can opt into uvloop for an extra speed bump: Requires Python 3.10+. Quickstart Run it: XPath parsers How it works 1. Spider kicks off from start url under a concurrency budget. 2. Parsers either follow (one argument) — discovering more URLs to queue — or extract (two arguments) — instantiating an Item from each matching page. 3. Items use Css / Xpath / Regex selectors to pull fields out of HTML. 4. save() is your async hook to persist results — write a file, push to a queue, insert into a database. Examples See the example/ directory for runnable scripts against Scrapinghub, V2EX, and Sciencenet. Development We use uv for packaging and ruff for lint + format. Install the pre-commit hooks: Contributing Pull requests are welcome. For non-trivial changes, please open an issue first to discuss. Make sure pytest and ruff check pass before submitting. License MIT © Elliot Gao","default_branch":null,"files":null,"tree":[],"storefront":"/r/elliotgao2","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/elliotgao2/gain/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}