{"repo":"zhuyingda/webster","free":true,"listed":false,"github":"https://github.com/zhuyingda/webster","clone":"git clone https://github.com/zhuyingda/webster.git","description":"a reliable high-level web crawling & scraping framework for Node.js.","language":"JavaScript","stars":559,"topics":["scraping-framework","crawler","crawling","headless-chrome","chromium","spider","automation-ui","automation-test","nodejs","nodejs-framework"],"license":"GPL-3.0","category":"scrapers-browser-automation","readme_excerpt":"Webster Overview Webster is a reliable web crawling and scraping framework written with Node.js, used to crawl websites and extract structured data from their pages. Which is different from other crawling framework is that Webster can scrape the content which rendered by browser client side javascript and ajax request Quick Start Let's start a simple crawler request to google website: Requirements - Node.js 10.x+ - Works on Linux, Mac OSX Or you can deploy on Docker. Install Single spider example Docker cluster example Pull the example docker image: In this docker image, there is a simple cluster-able example: You can organize your crawler cluster by Consumer and Producer like this: Usage on Raspbian Platform Documentation You can see more details from here. Code Contributors This project exists thanks to all the people who contribute. [Contribute]. Financial Contributors Become a financial contributor and help us sustain our community. [Contribute] Individuals Organizations Support this project with your organization. Your logo will show up here with a link to your website. [Contribute] License GPL-V3 Copyright (c) 2017-present, Yingda (Sugar) Zhu","default_branch":null,"files":null,"tree":[],"storefront":"/r/zhuyingda","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/zhuyingda/webster/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}