{"repo":"IonicaBizau/scrape-it","free":true,"listed":false,"github":"https://github.com/IonicaBizau/scrape-it","clone":"git clone https://github.com/IonicaBizau/scrape-it.git","description":"🔮 A Node.js scraper for humans.","language":"JavaScript","stars":4076,"topics":["scraper","node-scraper","hacktoberfest"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"scrape-it [![Support me on Patreon][badge patreon]][patreon] [![Buy me a book][badge amazon]][amazon] [![PayPal][badge paypal donate]][paypal-donations] A Node.js scraper for humans. ---- Sponsored with :heart: by: SerpApi.com Talordata SERP API: Real-time, accurate search data to power every decision Rapidproxy——Power your data collection and automation with 90M+ reliable residential proxies built for scale. :cloud: Installation :bulb: ProTip : You can install the cli version of this module by running npm install --global scrape-it-cli (or yarn global add scrape-it-cli ). FAQ Here are some frequent questions and their answers. 1. How to parse scrape pages? scrape-it has only a simple request module for making requests. That means you cannot directly parse ajax pages with it, but in general you will have those scenarios: 1. The ajax response is in JSON format. In this case, you can make the request directly, without needing a scraping library. 2. The ajax response gives you HTML back. Instead of calling the main website (e.g. example.com), pass to scrape-it the ajax url (e.g. example.com/api/that-endpoint ) and you will you will be able to parse the response 3. The ajax request is so complicated that you don't want to reverse-engineer it. In this case, use a headless browser (e.g. Google Chrome, Electron, PhantomJS) to load the content and then use the .scrapeHTML method from scrape it once you get the HTML loaded on the page. 2. Crawling There is no fancy way to crawl pages ","default_branch":null,"files":null,"tree":[],"storefront":"/r/IonicaBizau","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/IonicaBizau/scrape-it/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}