{"repo":"root4loot/recrawl","free":true,"listed":false,"github":"https://github.com/root4loot/recrawl","clone":"git clone https://github.com/root4loot/recrawl.git","description":"A web crawler written in Go","language":"Go","stars":10,"topics":["bugbounty","crawler","discovery","enumeration","go","golang","recon","reconnaissance","web"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"A web crawler written in Go Installation Go Docker Usage Parameter Mining When enabled with -mp/--mine-params , recrawl mines likely HTTP parameters from: - URL queries (e.g., ?q=term&page=2 ) - HTML forms ( , , ) - JavaScript bodies in fetch , XMLHttpRequest , and jQuery $.post/$.ajax - HTML data- attributes - Hidden inputs - GraphQL variable declarations - WebSocket URLs (query string) Parameters are grouped by certainty: high (URL/query + form fields), medium (JS/data/hidden/meta), low (reserved for weaker heuristics). Common non-parameters like class , style , etc. are ignored. Notes: - The parameter summary prints after the crawl finishes. - Parameter names must be simple identifiers (letters, numbers, , - ) and start with a letter or underscore. Web Discovery When enabled with -ed/--enable-discovery , recrawl performs web discovery fuzzing after normal crawling. Uses a curated 1022-entry wordlist containing: - Common files and directories (from SecLists raft-small) - API endpoints ( api/ , api/auth , etc.) - NPM/dev paths ( package.json , node modules , .env , etc.) Scope Behavior - If no includes are defined, everything is in scope unless explicitly excluded. - If includes are defined, only those targets are in scope; everything else is out of scope by default. - Exclusions always take priority over inclusions. Example Example running Running recrawl against hackerone.com to filter JavaScript files: Other ways to set target Pipe the target URL Pipe a file containing ta","default_branch":null,"files":null,"tree":[],"storefront":"/r/root4loot","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/root4loot/recrawl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}