{"repo":"roys/cewler","free":true,"listed":false,"github":"https://github.com/roys/cewler","clone":"git clone https://github.com/roys/cewler.git","description":"CeWLeR - Custom Word List generator Redefined. CeWL alternative in Python, based on the Scrapy framework.","language":"Python","stars":161,"topics":["bugbounty","crawler","reconnaissance","spider"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"CeWLeR - Custom Word List generator Redefined CeWLeR crawls from a specified URL and collects words to create a custom wordlist. It's a great tool for security testers and bug bounty hunters. The lists can be used for password cracking, subdomain enumeration, directory and file brute forcing, API endpoint discovery, etc. It's good to have an additional target specific wordlist that is different than what everybody else use. CeWLeR was sort of originally inspired by the really nice tool CeWL. I had some challenges with CeWL on a site I wanted a wordlist from, but without any Ruby experience I didn't know how to contribute or work around it. So instead I created a custom wordlist generator in Python to get the job done. At a glance Features - Generates custom wordlists by scraping words from web sites - A lot of options: - Output to screen or file - Can stay within subdomain, or visit sibling and child subdomains, or visit anything within the same top domain - Can stay within a certain depth of a website - Speed can be controlled - Word length and casing can be configured - JavaScript and CSS can be included - Text can be extracted from PDF files (using pypdf) - Crawled URLs can be output to separate file - Scraped e-mail addresses can also be output to separate file - Custom HTTP headers can be added - ++ - Using the excellent Scrapy framework for scraping and using the beautiful rich library for terminal output Commands and options Quick examples Output to file Will output to","default_branch":null,"files":null,"tree":[],"storefront":"/r/roys","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/roys/cewler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}