{"repo":"EasyPi/docker-scrapyd","free":true,"listed":false,"github":"https://github.com/EasyPi/docker-scrapyd","clone":"git clone https://github.com/EasyPi/docker-scrapyd.git","description":"🕷️ Scrapyd is an application for deploying and running Scrapy spiders.","language":"Dockerfile","stars":87,"topics":["scrapy","scrapyd","docker"],"license":"MIT","category":"deployment-docker-iac","readme_excerpt":"scrapyd ======= [scrapy][1] is an open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way. [scrapyd][2] is a service for running Scrapy spiders. It allows you to deploy your Scrapy projects and control their spiders using a HTTP JSON API. [scrapyd-client][3] is a client for scrapyd. It provides the scrapyd-deploy utility which allows you to deploy your project to a Scrapyd server. [scrapy-splash][4] provides Scrapy+JavaScript integration using Splash. [scrapyrt][5] allows you to easily add HTTP API to your existing Scrapy project. [spidermon][6] is a framework to build monitors for Scrapy spiders. [scrapy-poet][7] is the web-poet Page Object pattern implementation for Scrapy. [scrapy-playwright][8] is a Scrapy Download Handler which performs requests using Playwright for Python. This image is based on debian:bookworm , 8 latest stable python packages are installed: - scrapy==2.17.0 - scrapyd==1.6.0 - scrapyd-client==2.0.3 - scrapy-splash==0.11.1 - scrapyrt==v0.18.1 - spidermon==1.27.0 - scrapy-poet==0.27.2 - scrapy-playwright==v0.0.48 Please use this as base image for your own project. :warning: Scrapy (since [2.0.0][9]) has dropped support for Python 2.7, which reached end-of-life on 2020-01-01. docker-compose.yml Run it as background-daemon for scrapyd File: scrapy.cfg Run it as interactive-shell for scrapy Run it as realtime crawler for scrapyrt [1]: https://github.com/scrapy/scrapy [2]: https://github.c","default_branch":null,"files":null,"tree":[],"storefront":"/r/EasyPi","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/EasyPi/docker-scrapyd/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}