{"repo":"zytedata/zyte-smartproxy-headless-proxy","free":true,"listed":false,"github":"https://github.com/zytedata/zyte-smartproxy-headless-proxy","clone":"git clone https://github.com/zytedata/zyte-smartproxy-headless-proxy.git","description":"A complimentary proxy to help to use SPM with headless browsers","language":"Go","stars":109,"topics":["scraping","proxy","crawler"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Crawlera Headless Proxy Crawlera Headless proxy is a proxy which main intent is to help users with headless browsers to use Crawlera. This includes different implementations of headless browsers such as Splash, headless Chrome, and Firefox. Also, this proxy should help users of such frameworks as Selenium and Puppeteer to use Crawlera without a need to build Squid chains or install Polipo. The biggest problem with headless browsers is their configuration: 1. Crawlera uses proxy authentication protocol described in RFC 7235 but it is rather hard to configure such authentication in headless browsers. The most popular way of bypassing this problem is to use Polipo which is, unfortunately, unsupported for a long time. 2. Crawlera uses X-Headers as configuration. To use this API with headless browsers, users have to install plugins or extensions in their browsers and configure them to propagate such headers to Crawlera. 3. Also, it is rather hard and complex to maintain best practices of using these headers. For example, support of Browser Profiles requires to have a minimal possible set of headers. For example, it is recommended to remove Accept header by default. It is rather hard to do that using headless browsers API. 4. Crawlera works best with browsers only with some preconditions which users have to repeat everytime: session usage, some recommended headers like Referer etc. Crawlera Headless Proxy intended to help users to avoid such problems. You should generally think abo","default_branch":null,"files":null,"tree":[],"storefront":"/r/zytedata","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/zytedata/zyte-smartproxy-headless-proxy/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}