{"repo":"jkelin/forward-proxy-manager","free":true,"listed":false,"github":"https://github.com/jkelin/forward-proxy-manager","clone":"git clone https://github.com/jkelin/forward-proxy-manager.git","description":"Request distributor for web scraping","language":"TypeScript","stars":14,"topics":["proxy","proxy-server","scraper","webscraping"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"Forward Proxy Manager A proxy server that distributes your requests over a pool of proxies. It is made to easily scrape massive amounts of pages on websites behind rate-limiters and other systems without expensive bot protection. Mainly Cloudflare on normal settings. It makes individual requests over random proxies from the pool, so it is best suited for unauthenticated sessions and public pages. Features - Fetches and validates proxy list from a given URL - Fakes user agent string and some headers to avoid bot detection - Interacts via HTTPS PROXY protocol for seamless integration with various HTTP clients. - Also has a gRPC interface for more advanced use cases - Supports HTTPS, HTTP2, persistent connections for high performance - Keeps track of rate limits on individual proxy-target pairs and backs off on 429 (Too Many Requests) errors - Retry mechanism for failed requests using alternative proxies - Forwards most headers from client to target - Adjustable request priority using x-priority header - Built-in request queue for bulk requests without rate limit concerns - Optional web dashboard for real-time monitoring of pending requests Usage 1. Set Up the Proxy Manager With Docker: Using docker-compose: 2. Integrate with HTTP Client Proxy Manager is a MITM proxy and uses self-signed certificate. You must configure your HTTP client to ignore certificate errors. For Got in Node.js: Note: Since this proxy uses a self-signed certificate, you'll need to configure the HTTP client","default_branch":null,"files":null,"tree":[],"storefront":"/r/jkelin","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jkelin/forward-proxy-manager/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}