{"repo":"greenpeace/gpes-check-my-pages","free":true,"listed":false,"github":"https://github.com/greenpeace/gpes-check-my-pages","clone":"git clone https://github.com/greenpeace/gpes-check-my-pages.git","description":"Scrapping script used to test the Spanish web archive and redirects system, with more than 10,000 pages. It checks redirections, http responses, analytics, files hosted in soon-to-die servers, canonical urls and more.","language":"Go","stars":46,"topics":["golang","command-line-tool","csv","scraper"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"check-my-pages check-my-pages is a scraping script to test large sites. We are using it to test the Spanish soon-to-be archived site and redirects system, with more than 10,000 pages. It checks redirections, http responses, analytics, files hosted in soon-to-die servers, canonical urls and more. It checks each url in a list and creates report files about what was tested. Each file reports about a specific issue and includes the scanned url together with the result. Please note that the script was written to check for problems detected when manually inspecting a small sample (about 1%) of the Spanish website. This script was not built to be used as-it-is in other websites, you are supposed to modify it. This command line script complements other command line tools like ack, grep, rpl and others. Our own ecounter and ecompare are also used. Download and install Install the latest version Go to the releases page and download the last version for your 64bit operating system: Windows, Mac and Linux. Install from source You need Go 1.26.5 or newer. Then you can download and install it with: This installs a binary named gpes-check-my-pages in your Go bin directory. Or clone this repository and build the check-my-pages binary yourself: Build for Windows, Mac and Linux Go can cross-compile for other operating systems without extra tools: Update from source Get help If you downloaded the latest version from the releases page or built it with go build do: If you installed it with go ins","default_branch":null,"files":null,"tree":[],"storefront":"/r/greenpeace","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/greenpeace/gpes-check-my-pages/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}