{"repo":"Insutanto/scrapy-distributed","free":true,"listed":false,"github":"https://github.com/Insutanto/scrapy-distributed","clone":"git clone https://github.com/Insutanto/scrapy-distributed.git","description":"A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.","language":"Python","stars":62,"topics":["rabbitmq","scrapy","rabbitmq-pipeline","kafka","redisbloom","redis","distributed-spider","spider","python","crawler"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"Scrapy-Distributed Scrapy-Distributed is a series of components for you to develop a distributed crawler base on Scrapy in an easy way. Now! Scrapy-Distributed has supported RabbitMQ Scheduler , Kafka Scheduler and RedisBloom DupeFilter . You can use either of those in your Scrapy's project very easily. Features - RabbitMQ Scheduler - Support custom declare a RabbitMQ's Queue. Such as passive , durable , exclusive , auto delete , and all other options. - RabbitMQ Pipeline - Support custom declare a RabbitMQ's Queue for the items of spider. Such as passive , durable , exclusive , auto delete , and all other options. - Kafka Scheduler - Support custom declare a Kafka's Topic. Such as num partitions , replication factor and will support other options. - RedisBloom DupeFilter - Support custom the key , errorRate , capacity , expansion and auto-scaling( noScale ) of a bloom filter. - Custom DupeFilter Interface - Implement your own deduplication logic by extending BaseDupeFilter . Requirements - Python = 3.6 - Scrapy = 1.8.0 - Pika = 1.0.0 - RedisBloom = 0.2.0 - Redis = 3.0.1 - kafka-python = 1.4.7 TODO - RabbitMQ Item Pipeline - Support Delayed Message in RabbitMQ Scheduler - Support Scheduler Serializer - Custom Interface for DupeFilter - RocketMQ Scheduler - RocketMQ Item Pipeline - SQLAlchemy Item Pipeline - Mongodb Item Pipeline - Kafka Scheduler - Kafka Item Pipeline Usage Step 0: OR There are example projects in examples/rabbitmq example and examples/kafka example . Here is","default_branch":null,"files":null,"tree":[],"storefront":"/r/Insutanto","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Insutanto/scrapy-distributed/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}