{"repo":"fake-name/ReadableWebProxy","free":true,"listed":false,"github":"https://github.com/fake-name/ReadableWebProxy","clone":"git clone https://github.com/fake-name/ReadableWebProxy.git","description":"Rewriting web proxy and archival tool. At this point, it just tries to download all the things.","language":"Python","stars":204,"topics":["python","web-proxy","scraper"],"license":"BSD-3-Clause","category":"scrapers-browser-automation","readme_excerpt":"Readable-Web Proxy Reading long-form content on the internet is a shitty experience. This is a web-proxy that tries to make it better. This is a rewriting proxy . In other words, it proxies arbitrary web content, while allowing the rewriting of the remote content as driven by a set of rule-files. The goal is to effectively allow the complete customization of any existing web-sites as driven by predefined rules. Functionally, it's used for extracting just the actual content body of a site and reproducing it in a clean layout. It also modifies all links on the page to point to internal addresses, so following a link points to the proxied version of the file, rather then the original. --- While the above was the original scope, the project has mutated heavily. At this point, it has a complete web spider and archives entire websites to local storage. Additionally, multiple versions of each page are kept, with a overall rolling refresh of the entire database at configurable intervals (configurable on a per-domain, or global basis). There are also a lot of facilities responsible for feeding the releases/RSS views as part of wlnupdates.com. --- Quick installation overview: - Install Redis - (optional) install InfluxDB - (optional) install Graphite - Install Postgresql = 10. - Build the community extensions for Postgresql. - Create a database for the project. - In the project database, install the pg trgm and citext extensions from the community extensions modules. - Copy settings.ex","default_branch":null,"files":null,"tree":[],"storefront":"/r/fake-name","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/fake-name/ReadableWebProxy/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}