{"repo":"cyclone-github/spider","free":true,"listed":false,"github":"https://github.com/cyclone-github/spider","clone":"git clone https://github.com/cyclone-github/spider.git","description":"Spider - web crawler and local wordlist processor to generate frequency sorted wordlist / ngrams","language":"Go","stars":28,"topics":["cewl","crawler","cyclone","generator","n-gram","ngram","scaping","scraper","spider","url"],"license":"GPL-2.0","category":"scrapers-browser-automation","readme_excerpt":"Spider is a web crawler and wordlist/ngram generator written in Go that crawls specified URLs or local files to produce frequency-sorted wordlists and ngrams. Users can customize crawl depth, output files, frequency sort, and ngram options, making it ideal for web scraping to create targeted wordlists for tools like hashcat or John the Ripper. Since Spider is written in Go, it requires no additional libraries to download or install. Spider just works . Install latest release: Install from latest source code (bleeding edge): Modes - URL mode ( -url ) — crawl a website and create wordlist/ngrams (frequency sorted optional) - File mode ( -file ) — process a local text file to create wordlist/ngrams (frequency sorted optional) Spider: URL Mode When -text-match is used, all pages are still crawled for URL discovery but only pages with matching text are added to the wordlist. Crawl progress shows scanned vs matched: Spider: File Mode Wordlist & ngram creation tool to crawl a given url or process a local file to create wordlists and/or ngrams (depending on flags given). Usage Instructions: - To create a simple wordlist from a specified url (will save deduplicated wordlist to url spider.txt if -o is not set): - spider -url 'https://github.com/cyclone-github' - To set url crawl depth of 2 and create ngrams len 1-5, use flag \"-crawl 2\" and \"-ngram 1-5\" - spider -url 'https://github.com/cyclone-github' -crawl 2 -ngram 1-5 - To set a custom output file, use flag \"-o filename\" - spider -u","default_branch":null,"files":null,"tree":[],"storefront":"/r/cyclone-github","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/cyclone-github/spider/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}