{"repo":"ihandmine/aioscpy","free":true,"listed":false,"github":"https://github.com/ihandmine/aioscpy","clone":"git clone https://github.com/ihandmine/aioscpy.git","description":"An asyncio + aiolibs crawler imitate scrapy framework","language":"Python","stars":115,"topics":["asyncio","framework","python3","scrapy","aiohttp","loguru","scrapy-redis","crawling"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Aioscpy A powerful, high-performance asynchronous web crawling and scraping framework built on Python's asyncio ecosystem. English 中文 Overview Aioscpy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It draws inspiration from Scrapy and scrapy redis but is designed from the ground up to leverage the full power of asynchronous programming. Key Features - Fully Asynchronous : Built on Python's asyncio for high-performance concurrent operations - Scrapy-like API : Familiar API for those coming from Scrapy - Distributed Crawling : Support for distributed crawling using Redis - Multiple HTTP Backends : Support for aiohttp, httpx, and requests - Dynamic Variable Injection : Powerful dependency injection system - Flexible Middleware System : Customizable request/response processing pipeline - Robust Item Processing : Pipeline for processing scraped items Requirements - Python 3.8+ - Works on Linux, Windows, macOS, BSD Installation Basic Installation With All Dependencies With Specific HTTP Backends Latest Version from GitHub Quick Start Creating a New Project Creating a Spider This will create a basic spider in the spiders directory. Example Spider Creating a Single Spider Script Advanced Spider Example Running Spiders Running from Python Code Configuration Aioscpy can be configured through the settings.py file in your project. Here are the most important settings: Concurrency Settings Download Setting","default_branch":null,"files":null,"tree":[],"storefront":"/r/ihandmine","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ihandmine/aioscpy/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}