{"repo":"nfx/slrp","free":true,"listed":false,"github":"https://github.com/nfx/slrp","clone":"git clone https://github.com/nfx/slrp.git","description":"rotating open proxy multiplexer","language":"Go","stars":199,"topics":["crawler","proxy-pool","golang","proxy","proxy-checker","proxy-list","proxy-server"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"SLRP - rotating open proxy multiplexer --- Searches for proxies in open sources Intelligently stores state on disk across restarts Validates via configurable speed thresholds and anonymity Multiplexes HTTP/HTTPS MITM to HTTP, HTTPS, SOCKS4, and SOCKS5 Exposes REST API for refresh stats and pool health Exposes minimal Query Language for filtering of History and Proxy Stats. Records request history in-memory for further UI inspection Real-time statistics display about available pool Packaged as a single executable binary, that also includes Web UI Usage Download service, start it up, wait couple of minutes for the pool to pick up. Now run curl --proxy-insecure -D - -x http://127.0.0.1:8090 -k http://httpbin.org/get couple of times and see different origins and user agent headers. Concepts Source is an async process that looks at one or more pages for refreshed proxy list. Refresher component does best effort on scheduling items. Some sources perform better forwarded through a Pool , warming it up. One proxy may be seen in multiple sources , so we keep exclusive proxies per source across refreshes, which are not found in other sources. Proxy consists of protocol (HTTP, HTTPS, SOCKS4, or SOCKS5) and IP:PORT. Proxy becomes Scheduled immediately after it's seen in the source. Scheduled could transition into Probing queue if it's not Ignored (e.g. Timeouts or Blacklist ). Probing uses configurable pool of rotating anonymity checkers to check for liveliness. Timeout items are re-adde","default_branch":null,"files":null,"tree":[],"storefront":"/r/nfx","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/nfx/slrp/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}