{"repo":"INNOVINATI/microwler","free":true,"listed":false,"github":"https://github.com/INNOVINATI/microwler","clone":"git clone https://github.com/INNOVINATI/microwler.git","description":"A micro-framework for asynchronous deep crawls and web scraping with Python","language":"Python","stars":13,"topics":["web-crawling","web-scraping","micro-framework","python","asyncio","aiohttp","parsel","quart","nuxt"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Maintained as a solo-developer FOSS project. Small, reviewable pull requests are welcome. Microwler is a micro-framework for asynchronous web crawling and scraping implemented in Python. It is designed for deep crawls, i.e. retrieving every page of a specific website. Core features: - define crawlers in a single file using a declarative approach - control your crawlers via CLI , Python scripts or HTTP - cache your results on disk - use incremental crawling mode (only crawl new pages) - use pre-built selectors to scrape common data, i.e. or - easily define custom selectors using XPath & CSS - define \"transformers\" to manipulate scraped data after crawling - export data with pre-built plugins or build your own - use the built-in web service (API + Frontend) Sounds good? Head over to the Getting Started guide now! Microwler targets Python 3.12+ . If you're familiar with Microwler , you can install the package from PyPI using uv : Inspiration Many concepts were inspired by experience with Scrapy , the industry standard for web crawling with Python. The idea was to design a very minimalistic framework for analysing specific websites. We tried to focus on making things as simple as possible for developers. With Microwler, you can build crawlers and extract relevant data in a breeze - and in a single file. Contributing Contributions are welcome via short-lived branches and pull requests to main . For local setup, hooks, CI expectations, conventional commits, and the release workflow","default_branch":null,"files":null,"tree":[],"storefront":"/r/INNOVINATI","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/INNOVINATI/microwler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}