{"repo":"hueristiq/xcrawl3r","free":true,"listed":false,"github":"https://github.com/hueristiq/xcrawl3r","clone":"git clone https://github.com/hueristiq/xcrawl3r.git","description":"A command-line utility designed to recursively spider webpages for URLs. It works by actively traversing websites - following links embedded in webpages, parsing resources like sitemaps and robots.txt files, and even processing local files - to uncover every URL.","language":"Go","stars":111,"topics":["crawler","bug-bounty","penetration-testing","penetration-testing-tools","contentdiscovery","reconnaissance","bug-bounty-tools","ethical-hacking","ethical-hacking-tools","red-teaming"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"xcrawl3r xcrawl3r is a command-line utility designed to recursively spider webpages for URLs. It works by actively traversing websites - following links embedded in webpages, and parsing files (including sitemaps & robots.txt) - to uncover every URL. Unlike xurlfind3r that doesn't interact directly with the target, xcrawl3r interacts directly with the target by spidering its pages in real time. This active approach allows it to discover URLs that may be hidden or unindexed, providing a complete picture of the website’s navigational flow and content distribution. This makes xcrawl3r a powerful tool for security researchers, IT professionals, and anyone looking to gain insights into the URLs associated with websites. Resources - Features - Installation - Install release binaries (Without Go Installed) - Install source (With Go Installed) - go install ... - go build ... the development version - Install on Docker (With Docker Installed) - Post Installation - Usage - Contributing - Licensing Features - Recursively spiders webpages for URLs - Extracts URLs from files (including sitemaps & robots.txt ) - Supports stdin and stdout for easy integration in automated workflows - Supports multiple output formats (JSONL, file, stdout) - Cross-Platform (Windows, Linux & macOS) Installation Install release binaries (Without Go Installed) Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its UR","default_branch":null,"files":null,"tree":[],"storefront":"/r/hueristiq","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/hueristiq/xcrawl3r/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}