{"repo":"clemfromspace/scrapy-selenium","free":true,"listed":false,"github":"https://github.com/clemfromspace/scrapy-selenium","clone":"git clone https://github.com/clemfromspace/scrapy-selenium.git","description":"Scrapy middleware to handle javascript pages using selenium","language":"Python","stars":952,"topics":["scrapy","selenium","crawling"],"license":"WTFPL","category":"scrapers-browser-automation","readme_excerpt":"Scrapy with selenium Scrapy middleware to handle javascript pages using selenium. Installation You should use python =3.6 . You will also need one of the Selenium compatible browsers. Configuration 1. Add the browser to use, the path to the driver executable, and the arguments to pass to the executable to the scrapy settings: Optionally, set the path to the browser executable: In order to use a remote Selenium driver, specify SELENIUM COMMAND EXECUTOR instead of SELENIUM DRIVER EXECUTABLE PATH : 2. Add the SeleniumMiddleware to the downloader middlewares: Usage Use the scrapy selenium.SeleniumRequest instead of the scrapy built-in Request like below: The request will be handled by selenium, and the request will have an additional meta key, named driver containing the selenium driver with the request processed. For more information about the available driver methods and attributes, refer to the selenium python documentation The selector response attribute work as usual (but contains the html processed by the selenium driver). Additional arguments The scrapy selenium.SeleniumRequest accept 4 additional arguments: wait time / wait until When used, selenium will perform an Explicit wait before returning the response to the spider. screenshot When used, selenium will take a screenshot of the page and the binary data of the .png captured will be added to the response meta : script When used, selenium will execute custom JavaScript code.","default_branch":null,"files":null,"tree":[],"storefront":"/r/clemfromspace","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/clemfromspace/scrapy-selenium/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}