{"repo":"hakluke/hakrawler","free":true,"listed":false,"github":"https://github.com/hakluke/hakrawler","clone":"git clone https://github.com/hakluke/hakrawler.git","description":"Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application","language":"Go","stars":5110,"topics":["bugbounty","crawling","hacking","osint","pentesting","recon","reconnaissance"],"license":"GPL-3.0","category":"security-tools","readme_excerpt":"Hakrawler Fast golang web crawler for gathering URLs and JavaScript file locations. This is basically a simple implementation of the awesome Gocolly library. Example usages Single URL: Multiple URLs: Timeout for each line of stdin after 5 seconds: Send all requests through a proxy: Include subdomains: Note: a common issue is that the tool returns no URLs. This usually happens when a domain is specified (https://example.com), but it redirects to a subdomain (https://www.example.com). The subdomain is not included in the scope, so the no URLs are printed. In order to overcome this, either specify the final URL in the redirect chain or use the -subs option to include subdomains. Example tool chain Get all subdomains of google, find the ones that respond to http(s), crawl them all. Installation Normal Install First, you'll need to install go. Then run this command to download + compile hakrawler: You can now run /go/bin/hakrawler . If you'd like to just run hakrawler without the full path, you'll need to export PATH=\" /go/bin/:$PATH\" . You can also add this line to your /.bashrc file if you'd like this to persist. Docker Install (from dockerhub) Local Docker Install It's much easier to use the dockerhub method above, but if you'd prefer to run it locally: Kali Linux: Using apt Note: This will install an older version of hakrawler without all the features, and it may be buggy. I recommend using one of the other methods. Then, to run hakrawler: Command-line options --- Built by hak","default_branch":null,"files":null,"tree":[],"storefront":"/r/hakluke","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/hakluke/hakrawler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}