{"repo":"scrapy-plugins/scrapy-playwright","free":true,"listed":false,"github":"https://github.com/scrapy-plugins/scrapy-playwright","clone":"git clone https://github.com/scrapy-plugins/scrapy-playwright.git","description":"🎭 Playwright integration for Scrapy","language":"Python","stars":1440,"topics":["playwright","playwright-python","scrapy","python","python3","python-asyncio","javascript-renderer","headless-browser","chrome-headless","firefox-headless"],"license":"BSD-3-Clause","category":"scrapers-browser-automation","readme_excerpt":"scrapy-playwright: Playwright integration for Scrapy A Scrapy Download Handler which performs requests using Playwright for Python. It can be used to handle pages that require JavaScript (among other things), while adhering to the regular Scrapy workflow (i.e. without interfering with request scheduling, item processing, etc). Requirements After the release of version 2.0, which includes coroutine syntax support and asyncio support, Scrapy allows to integrate asyncio -based projects such as Playwright . Minimum required versions Python = 3.10 Scrapy = 2.7 Playwright = 1.40 Installation scrapy-playwright is available on PyPI and can be installed with pip : playwright is defined as a dependency so it gets installed automatically, however it might be necessary to install the specific browser(s) that will be used: It's also possible to install only a subset of the available browsers: Changelog See the changelog document. Activation Download handler Replace the default https and/or http Download Handlers through DOWNLOAD HANDLERS : Registering the handler for https is usually enough, since most modern sites use HTTPS. Enable the http handler only if you need to process plain HTTP URLs with Playwright. Note that the ScrapyPlaywrightDownloadHandler class inherits from the default http/https handler. Unless explicitly marked (see Basic usage), requests will be processed by the regular Scrapy download handler. Twisted reactor Install the asyncio -based Twisted reactor: This is the def","default_branch":null,"files":null,"tree":[],"storefront":"/r/scrapy-plugins","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/scrapy-plugins/scrapy-playwright/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}