{"repo":"alephdata/memorious","free":true,"listed":false,"github":"https://github.com/alephdata/memorious","clone":"git clone https://github.com/alephdata/memorious.git","description":"Lightweight web scraping toolkit for documents and structured data.","language":"Python","stars":316,"topics":["crawling","scraping","scraping-framework"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Memorious Notice: The project is no longer maintained. :warning: We recommend using memorious4 fork that DARC has been actively maintaining further on. The solitary and lucid spectator of a multiform, instantaneous and almost intolerably precise world. -- Funes the Memorious, Jorge Luis Borges memorious is a light-weight web scraping toolkit. It supports scrapers that collect structured or un-structured data. This includes the following use cases: - Make crawlers modular and simple tasks re-usable - Provide utility functions to do common tasks such as data storage, HTTP session management - Integrate crawlers with the Aleph and FollowTheMoney ecosystem - Get out of your way as much as possible Design When writing a scraper, you often need to paginate through through an index page, then download an HTML page for each result and finally parse that page and insert or update a record in a database. memorious handles this by managing a set of crawlers , each of which can be composed of multiple stages . Each stage is implemented using a Python function, which can be re-used across different crawlers . The basic steps of writing a Memorious crawler: 1. Make YAML crawler configuration file 2. Add different stages 3. Write code for stage operations (optional) 4. Test, rinse, repeat Documentation The documentation for Memorious is available at alephdata.github.io/memorious. Feel free to edit the source files in the docs folder and send pull requests for improvements. To build the docu","default_branch":null,"files":null,"tree":[],"storefront":"/r/alephdata","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/alephdata/memorious/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}