{"repo":"groupbwt/scrapy-boilerplate","free":true,"listed":false,"github":"https://github.com/groupbwt/scrapy-boilerplate","clone":"git clone https://github.com/groupbwt/scrapy-boilerplate.git","description":"Scrapy project boilerplate done right","language":"Python","stars":48,"topics":["scrapy","boilerplate","python"],"license":"MIT","category":"saas-starters-boilerplates","readme_excerpt":"scrapy-boilerplate This is a boilerplate for new Scrapy projects. The project is a WIP, so expect major changes and additions (mostly latter). Master branch is to be considered as always ready to use, with major changes/features introduced in feature branches. Features - Python 3.11+ - Poetry for dependency management - SQLAlchemy ORM with alembic migrations - RabbitMQ integrated via pika - configuration via ENV variables and/or .env file - single file for each class - Docker-ready (see here) - PM2-ready - supports single-IP/rotating proxy config out of the box (see here) Installation Python Quickstart Guide To create and run a new Scrapy project using this boilerplate, you need to: 1. Clone the repository. 2. cp .env.example .env 3. No docker: 1. Have the following prerequisites: python 3.11+, poetry, mysqlclient libraries, etc 2. cd src/python/src 3. poetry install 4. poetry shell 5. scrapy 4. Docker: 1. Have the following prerequisites: docker, docker-compose 2. docker compose up -d database python 3. docker compose exec python bash 4. cd /var/app/python/src/ 5. poetry shell 6. scrapy Docker The project includes Dockerfiles and docker-compose configuration for running your spiders in containers. Also, a configuration for default RabbitMQ server is included. Dockerfiles are located inside the docker subdirectory, and the docker-compose.yml - at the root of the project. Docker-compose takes configuration values from ENV. Environment can also be provided by creating a .env fi","default_branch":null,"files":null,"tree":[],"storefront":"/r/groupbwt","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/groupbwt/scrapy-boilerplate/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}