{"repo":"ispras/scrapy-puppeteer","free":true,"listed":false,"github":"https://github.com/ispras/scrapy-puppeteer","clone":"git clone https://github.com/ispras/scrapy-puppeteer.git","description":"Library that helps use puppeteer in scrapy.","language":"Python","stars":51,"topics":["scrapy-puppeteer","middleware","spider","scrapy"],"license":"BSD-3-Clause","category":"scrapers-browser-automation","readme_excerpt":"Scrapy-puppeteer-client This package aims to manage Chrome browser with Puppeteer from Scrapy spiders. This allows to scrape sites that require JS to function properly and to make the scraper more similar to humans. It is a client library for scrapy-puppeteer-service. ⚠️ This repository is under development. This project is under development. Use it at your own risk. Installation Using pip (master branch): Configuration You should have scrapy-puppeteer-service started. Then add its URL to settings.py and enable puppeteer downloader middleware: Available methods: Puppeteer , Pyppeteer , Playwright Pyppeteer and Playwright methods do not require a running service. They use the pyppeteer and playwright libraries for Python to interact with the browser. Actions such as CustomJsAction , RecaptchaSolver , and Har are not available when using these methods. To use Pyppeteer or Playwright methods you need to install Chromium. Basic usage Use scrapypuppeteer.PuppeteerRequest instead of scrapy.Request to render URLs with Puppeteer: Puppeter responses There is a parent PuppeteerResponse class from which other response classes are inherited. Here is a list of them all: - PuppeteerHtmlResponse - has html and cookies properties - PuppeteerScreenshotResponse - has screenshot property - PuppeteerHarResponse - has har property - PuppeteerJsonResponse - has data property and to html() method which tries to transform itself to PuppeteerHtmlResponse - PuppeteerRecaptchaSolverResponse(PuppeteerJs","default_branch":null,"files":null,"tree":[],"storefront":"/r/ispras","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ispras/scrapy-puppeteer/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}