{"repo":"miroshnikov/scrapyteer","free":true,"listed":false,"github":"https://github.com/miroshnikov/scrapyteer","clone":"git clone https://github.com/miroshnikov/scrapyteer.git","description":"Web crawling & scraping framework for Node.js on top of headless Chrome browser","language":"TypeScript","stars":20,"topics":["scrape","scraper","scrapy","scraping-websites","scrapy-crawler","crawer","web-crawler","web-scraping","scraping","crawling"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Scrapyteer Scrapyteer is a Node.js web scraping framework/tool/library built on top of the headless Chrome browser Puppeteer. It allows you to scrape both plain html pages and javascript generated content including SPAs (Single-Page Application) of any kind. Scrapyteer offers a small set of functions that forms an easy and concise DSL (Domain Specific Language) for web scraping and allows to define a crawling workflow and a shape of output data . - Examples - Installation - Configuration options - API Examples Scrapyteer uses a configuration file ( scrapyteer.config.js by default). Here are some examples: Simple example Search books on amazon.com and get titles and ISBNs of books on the first page of the results. More elaborate example Search books on amazon.com, get a number of attributes in JSON lines file and download the cover image of each book to a local directory. Installation Locally Locally as dependency in package.json : Globally Make sure $NODE PATH points to where global packages are located. If it doesn't, you may need to set it e.g. export NODE PATH=/path/to/global/node modules Configuration options save A file name or console object, by default output.json in the current directory. .json and .jsonl are currently supported. If format is json the data is first collected in memory and then dumped to the file in one go, in jsonl data is written line by line (good for large datasets). root The root URL to scrape parse The parsing workflow: a pipe function, an object","default_branch":null,"files":null,"tree":[],"storefront":"/r/miroshnikov","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/miroshnikov/scrapyteer/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}