{"repo":"ispras/scrapy-puppeteer-service","free":true,"listed":false,"github":"https://github.com/ispras/scrapy-puppeteer-service","clone":"git clone https://github.com/ispras/scrapy-puppeteer-service.git","description":"A special service that runs puppeteer instances.","language":"JavaScript","stars":18,"topics":["scrapy","puppeteer","selector"],"license":"BSD-3-Clause","category":"scrapers-browser-automation","readme_excerpt":"scrapy-puppeteer-service This is special service that runs puppeteer instances. It is a part of scrapy-puppeteer middleware that helps to handle javascript pages in scrapy using puppeteer. This allows to scrape sites that require JS to function properly and to make the scraper more similar to humans. ⚠️ This repository is under development. This project is under development. Use it at your own risk. Usage On your host machine you should enable user namespace cloning. To start service run the docker container. Since the Dockerfile adds a pptr user as a non-privileged user, it may not have all the necessary privileges. So you should use docker run --cap-add=SYS ADMIN option. To run example which shows how to deploy several instances of service with load balancer use this command. To run headfull puppeteer in container provide HEADLESS=false environment variable to the container. In this case a VNC server at localhost:5901 with password password is started. You may change the password providing VNC SERVER environment variable. API Here is the list of implemented methods that could be used to connect to puppeteer. For All requests puppeteer browser creates new incognito browser context and new page in it. If your want to reuse your browser context simple send context id in your query. All request return their context ids in response. Also, you could reuse your browser page and more actions with it. In order to do so you should send in your request pageId that is returned in your ","default_branch":null,"files":null,"tree":[],"storefront":"/r/ispras","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ispras/scrapy-puppeteer-service/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}