{"repo":"abo123456789/leek","free":true,"listed":false,"github":"https://github.com/abo123456789/leek","clone":"git clone https://github.com/abo123456789/leek.git","description":"Distributed task redisqueue(最简单python分布式函数调度框架)","language":"Python","stars":65,"topics":["redisqueue","queue-tasks","web-crawler","redis","thread-pool","producer-consumer","distribute-crawler","leek","kafka","sqlite3"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"中文文档 English Docs 任务发布消费中间件 功能描述 比scrapy更灵活,比celery更容易上手的分布式爬虫框架。用最少的代码,用最简单的方式,做最多的事情 1分钟内能熟练运用该框架爬取数据,无需学习复杂文档.轻松扩展各种中间件 特色说明： 支持多中间件： 支持reids kafka sqlite memory 四种中间件(首推redis,支持批量发布任务,分布式消费快如闪电) 并发支持： 支持process threading gevent三种并发消费模式(可混合使用) 控频限流： 精确控制1秒钟运行多少次函数 任务去重： 如果重复推送消费成功的任务,自动过滤掉该任务 消费确认： 启用消费确认,消费任务宕机手动终止情况,任务不会丢失 重试次数： 当函数运行出错，会立即重试指定的次数，达到最大次重试数后任务会进入死信队列 任务过期机制： 如果设定某个任务过期时间,任务超过设定过期时间还未消费,则会自动丢弃该任务 死信队列任务重新消费 进入死信队列任务支持手动重入队列重新消费 pip安装 1.发布任务和消费任务 2.发布任务和消费任务(更多参数实例) 3.发布任务和消费任务(装饰器版本) 消费函数参数详解 redisweb 通过浏览器查看任务消费情况","default_branch":null,"files":null,"tree":[],"storefront":"/r/abo123456789","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/abo123456789/leek/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}