{"repo":"cornelk/goscrape","free":true,"listed":false,"github":"https://github.com/cornelk/goscrape","clone":"git clone https://github.com/cornelk/goscrape.git","description":"Web scraper that can create an offline readable version of a website","language":"Go","stars":218,"topics":["scraper","golang","go","webscraping"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"goscrape - create offline browsable copies of websites A web scraper built with Golang. It downloads the content of a website and allows it to be archived and read offline. Features Features and advantages over existing tools like wget, httrack, Teleport Pro: Free and open source Available for all platforms that Golang supports JPEG and PNG images can be converted down in quality to save disk space Excluded URLS will not be fetched (unlike wget) No incomplete temp files are left on disk Downloaded asset files are skipped in a new scraper run Assets from external domains are downloaded automatically Sane default values Limitations No GUI version, console only Installation There are 2 options to install goscrape: 1. Download and unpack a binary release from Releases or 2. Compile the latest release from source: Compiling the tool from source code needs to have a recent version of Golang installed. Usage Scrape a website by running To serve the downloaded website directory in a local run webserver use Options Cookies Cookies can be passed in a file using the --cookiefile parameter and a file containing cookies in the following format: Proxy Configuration The --proxy flag supports multiple proxy protocols for scraping through different types of proxy servers: Supported Proxy Types - HTTP proxies : http://proxy.example.com:8080 - HTTPS proxies : https://proxy.example.com:8080 - SOCKS5 proxies : socks5://proxy.example.com:1080 Authentication Proxies requiring authentication can inc","default_branch":null,"files":null,"tree":[],"storefront":"/r/cornelk","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/cornelk/goscrape/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}