{"repo":"crawlab-team/crawlab","free":true,"listed":false,"github":"https://github.com/crawlab-team/crawlab","clone":"git clone https://github.com/crawlab-team/crawlab.git","description":"Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架","language":"Go","stars":12256,"topics":["webcrawler","scrapy","crawlab","spiders-management","go","scrapyd-ui","spider","crawler","webspider","web-crawler"],"license":"BSD-3-Clause","category":"scrapers-browser-automation","readme_excerpt":"Crawlab 中文 English Installation Run Screenshot Architecture Integration Compare Community & Sponsorship CHANGELOG Disclaimer Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium. Demo Documentation Installation You can follow the installation guide. Quick Start Please open the command line prompt and execute the command below. Make sure you have installed docker-compose in advance. Next, you can look into the docker-compose.yml (with detailed config params) and the Documentation for further information. Run Docker Please use docker-compose to one-click to start up. By doing so, you don't even have to configure MongoDB database. Create a file named docker-compose.yml and input the code below. Then execute the command below, and Crawlab Master and Worker Nodes + MongoDB will start up. Open the browser and enter http://localhost:8080 to see the UI interface. For Docker Deployment details, please refer to relevant documentation. Screenshot Login Home Page Node List Spider List Spider Overview Spider Files Task Log Task Results Cron Job Architecture The architecture of Crawlab is consisted of a master node, worker nodes, SeaweedFS (a distributed file system) and MongoDB database. The frontend app interacts with the master node, which communicates with other components such as MongoDB, SeaweedFS and worker nodes. Master node and worker n","default_branch":null,"files":null,"tree":[],"storefront":"/r/crawlab-team","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/crawlab-team/crawlab/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}