{"repo":"ConlinH/aio-scrapy","free":true,"listed":false,"github":"https://github.com/ConlinH/aio-scrapy","clone":"git clone https://github.com/ConlinH/aio-scrapy.git","description":"Implement scrapy with asyncio","language":"Python","stars":71,"topics":["aiohttp","crawler","httpx","scrapy","scrapy-redis","spider","aioscrapy","scrapyd"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"AioScrapy AioScrapy是一个基于Python异步IO的强大网络爬虫框架。它的设计理念源自Scrapy，但完全基于异步IO实现，提供更高的性能和更灵活的配置选项。 AioScrapy is a powerful asynchronous web crawling framework built on Python's asyncio library. It is inspired by Scrapy but completely reimplemented with asynchronous IO, offering higher performance and more flexible configuration options. 特性 Features - 完全异步 ：基于Python的asyncio库，实现高效的并发爬取 - 多种下载处理程序 ：支持多种HTTP客户端，包括aiohttp、httpx、requests、pyhttpx、curl cffi、DrissionPage、playwright和sbcdp - 灵活的中间件系统 ：轻松添加自定义功能和处理逻辑 - 强大的数据处理管道 ：支持多种数据库存储选项 - 内置信号系统 ：方便的事件处理机制 - 丰富的配置选项 ：高度可定制的爬虫行为 - 分布式爬取 ：支持使用Redis和RabbitMQ进行分布式爬取 - 数据库集成 ：内置支持Redis、MySQL、MongoDB、PostgreSQL和RabbitMQ - Fully Asynchronous : Built on Python's asyncio for efficient concurrent crawling - Multiple Download Handlers : Support for various HTTP clients including aiohttp, httpx, requests, pyhttpx, curl cffi, DrissionPage, playwright and sbcdp - Flexible Middleware System : Easily add custom functionality and processing logic - Powerful Data Processing Pipelines : Support for various database storage options - Built-in Signal System : Convenient event handling mechanism - Rich Configuration Options : Highly customizable crawler behavior - Distributed Crawling : Support for distributed crawling using Redis and RabbitMQ - Database Integration : Built-in support for Redis, MySQL, MongoDB, PostgreSQL, and RabbitMQ 安装 Installation 要求 Requirements - Python 3.9+ 使用pip安装 Install with pip 开始 Start 多版本测试 Multi-version Testing 项目使用tox验证Python 3.9至3.","default_branch":null,"files":null,"tree":[],"storefront":"/r/ConlinH","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ConlinH/aio-scrapy/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}