{"repo":"MikeChongCan/scylla","free":true,"listed":false,"github":"https://github.com/MikeChongCan/scylla","clone":"git clone https://github.com/MikeChongCan/scylla.git","description":"Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era","language":"Python","stars":4018,"topics":["crawler","python3","scylla","proxy-pool","python"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Scylla An intelligent proxy pool for humanities, to extract content from the internet and build your own Large Language Models in this new AI era. Key features: - Automatic proxy ip crawling and validation - Easy-to-use JSON API - Simple but beautiful web-based user interface (eg. geographical distribution of proxies) - Get started with only 1 command minimally - Simple HTTP Forward proxy server - [Scrapy] and [requests] integration with only 1 line of code minimally - Headless browser crawling Get started =========== Installation ------------ Install with Docker (highly recommended) Install directly via pip Install from source Usage ----- This is an example of running a service locally ( localhost ), using port 8899 . Note: You might have to wait for 1 to 2 minutes in order to get some proxy ips populated in the database for the first time you use Scylla. JSON API Proxy IP List Optional URL parameters: Parameters Default value Description ----------- ------------- ------------------------------------------------------------ page 1 The page number limit 20 The number of proxies shown on each page anonymous any Show anonymous proxies or not. Possible values： true , only anonymous proxies; false , only transparent proxies https any Show HTTPS proxies or not. Possible values： true , only HTTPS proxies; false , only HTTP proxies countries None Filter proxies for specific countries. Format example: US , or multi-countries: US,GB Sample result: System Statistics Sample result: HTTP","default_branch":null,"files":null,"tree":[],"storefront":"/r/MikeChongCan","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/MikeChongCan/scylla/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}