{"repo":"bitmakerla/estela","free":true,"listed":false,"github":"https://github.com/bitmakerla/estela","clone":"git clone https://github.com/bitmakerla/estela.git","description":"estela, an elastic web scraping cluster 🕸","language":"TypeScript","stars":201,"topics":["crawling","django","kubernetes","python","react","scraping","docker","hacktoberfest","python-requests","requests"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"estela estela is an elastic web scraping cluster running on Kubernetes. It provides mechanisms to deploy, run and scale web scraping spiders via a REST API and a web interface. Technologies Project Structure The project consists of three main modules: - REST API : built with the Django REST framework toolkit, it exposes several endpoints to manage projects, spiders, and jobs. It uses Celery for task processing and takes care of deploying your Scrapy projects, among other things. - Queueing : estela needs a high-throughput, low-latency platform that controls real-time data feeds in a producer-consumer architecture. In this module, you will find a consumer used to collect and transport the information from the spider jobs into a database. - Web : A web interface implemented with React and Typescript that lets you manage projects and spiders. Each of these modules works independently of the rest and can be changed. Each module has a more detailed description in its corresponding directory. estela-cli estela-cli is a command-line interface for estela. How to Contribute Please read CONTRIBUTING.md and follow the steps. Remember to abide by our adapted from ESTELA Code of Conduct too.","default_branch":null,"files":null,"tree":[],"storefront":"/r/bitmakerla","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/bitmakerla/estela/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}