{"repo":"q-m/scrapyd-k8s","free":true,"listed":false,"github":"https://github.com/q-m/scrapyd-k8s","clone":"git clone https://github.com/q-m/scrapyd-k8s.git","description":"Scrapyd on container infrastructure","language":"Python","stars":16,"topics":["kubernetes","scrapy","scrapyd","docker"],"license":"MIT","category":"deployment-docker-iac","readme_excerpt":"Scrapyd for Kubernetes Scrapyd-k8s is an application for deploying and running Scrapy spiders as either Docker instances or Kubernetes jobs. Its intention is to be compatible with scrapyd, but adapt to a container-based environment. There are some important differences, though: Spiders are distributed as Docker images , not as Python eggs. This allows to bundle spiders with dependencies, with all its benefits (and downsides). Each spider is run as a Docker instance or Kubernetes job , instead of a process. This gives good visibility within an already running cluster. Projects are specified in the configuration file , which means this can not be modified at run-time. On the other hand, scrapyd-k8s can be restarted without affecting any running spiders. At this moment, each spider job is directly linked to a Docker instance or Kubernetes job, and the daemon will retrieve its state by looking at those jobs. This makes it easy to inspect and adjust the spider queue even outside scrapyd-k8s. By default, there is no scheduling, and all jobs created will be started immediately. This can be changed by setting max proc . Running Typically this application will be run on using a (Docker or Kubernetes) container. You will need to provide a configuration file, use one of the sample configuration files as a template ( scrapyd k8s.sample-k8s.conf or scrapyd k8s.sample-docker.conf ). The next section explains how to get this running Docker, Kubernetes or Local. Then read on for an example o","default_branch":null,"files":null,"tree":[],"storefront":"/r/q-m","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/q-m/scrapyd-k8s/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}