{"repo":"mawrkus/jason-the-miner","free":true,"listed":false,"github":"https://github.com/mawrkus/jason-the-miner","clone":"git clone https://github.com/mawrkus/jason-the-miner.git","description":"⛏ A versatile Web scraper for Node.js","language":"JavaScript","stars":45,"topics":["scraping","crawling","scraper","crawler","web-scraper","javascript"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Jason the Miner Harvesting data at the mine... Here comes Jason the Miner, a versatile Web scraper for Node.js. ⛏ Features - Composable: via a modular architecture based on pluggable processors. The output of one processor feeds the input of the next one. There are 3 types of processors: 1. loaders: to fetch the data (via HTTP requests, by reading text files, etc.) 2. parsers: to parse the data (HTML by default) & extract the relevant parts according to a predefined schema 3. transformers: to transform and/or output the results (to a CSV file, via email, etc.) - Configurable: each processor can be chosen & configured independently - Extensible: you can register your own custom processors - CLI-friendly: Jason the Miner works well with pipes & redirections - Promise-based API - MIT-licensed ⛏ Installing If you don't want to install it, you can use it directly with npx: ⛏ Demos Clone the project: Then have a look at the demos folder, you'll find examples of scraping: - Simple GitHub search results (JSON, CSV, Markdown output) - More complex GitHub search results (including following links & paginating issues) - Goodreads books, following links to Amazon to grab their product ID - Google search results for finding mobile apps in various blogs, etc. - IMDb images gallery links with pagination - Mixcloud stats, templating them & sending them by mail - Mixcloud SPA scraping, by controlling a headless Chrome browser with Puppeteer - Avatars and downloading them - A CSV file to bulk ","default_branch":null,"files":null,"tree":[],"storefront":"/r/mawrkus","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mawrkus/jason-the-miner/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}