{"repo":"thomasdondorf/puppeteer-cluster","free":true,"listed":false,"github":"https://github.com/thomasdondorf/puppeteer-cluster","clone":"git clone https://github.com/thomasdondorf/puppeteer-cluster.git","description":"Puppeteer Pool, run a cluster of instances in parallel","language":"TypeScript","stars":3515,"topics":["puppeteer","pool","pooling","headless-chrome","cluster","node"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Puppeteer Cluster Create a cluster of puppeteer workers. This library spawns a pool of Chromium instances via [Puppeteer] and helps to keep track of jobs and errors. This is helpful if you want to crawl multiple pages or run tests in parallel. Puppeteer Cluster takes care of reusing Chromium and restarting the browser in case of errors. - Installation - Usage - Examples - Concurrency implementations - Typings for input/output (via TypeScript Generics) - Debugging - API - License What does this library do? Handling of crawling errors Auto restarts the browser in case of a crash Can automatically retry if a job fails Different concurrency models to choose from (pages, contexts, browsers) Simple to use, small boilerplate Progress view and monitoring statistics (see below) Installation Install using your favorite package manager: Alternatively, use yarn : Usage The following is a typical example of using puppeteer-cluster. A cluster is created with 2 concurrent workers. Then a task is defined which includes going to the URL and taking a screenshot. We then queue two jobs and wait for the cluster to finish. Examples Simple example Wait for a task to be executed Minimal screenshot server with express Deep crawling the Google search results Crawling the Alexa Top 1 Million Queuing functions (simple) Queuing functions (complex) Error handling Using a different puppeteer library (like puppeteer-core or puppeteer-firefox) Provide types for input/output with TypeScript generics Concurre","default_branch":null,"files":null,"tree":[],"storefront":"/r/thomasdondorf","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/thomasdondorf/puppeteer-cluster/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}