{"repo":"PadishahIII/SecretScraper","free":true,"listed":false,"github":"https://github.com/PadishahIII/SecretScraper","clone":"git clone https://github.com/PadishahIII/SecretScraper.git","description":"SecretScraper is a web scraper that crawl through target websites, scrape from http response and extract secret information via regular expression. Rust version: https://github.com/PadishahIII/secret-scraper-in-rust","language":"Python","stars":68,"topics":["crawler","cyper","hyperscan","python","sensitivity-analysis","webscraper","pentest-tool","pentesting"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"SecretScraper Overview SecretScraper is a highly configurable web scrape tool that crawl links from target websites and scrape sensitive data via regular expression. Feature - Web crawler: extract links via both DOM hierarchy and regex - Support domain white list and black list - Support multiple targets, input target URLs from a file - Support local file scan - Scalable customization: header, proxy, timeout, cookie, scrape depth, follow redirect, etc. - Built-in per-domain rate limiting and HTTP connection pool limits - Built-in regex to search for sensitive information - Flexible configuration in yaml format Prerequisite - Platform: Test on MacOS, Ubuntu and Windows. - Python Versions: 3.9 - 3.14 Usage Install Update Note that , since Secretscraper generates a default configuration under the work directory if settings.yml is absent, so remember to update the settings.yml to the latest version(just copy from Customize Configuration). Development Basic Usage Start with single target: Start with multiple targets: Sample output: All supported options: Advanced Usage Validate the Status of Links Use --validate option to check the status of found links, this helps reduce invalid links in the result. Thorough Crawl The max depth is set to 1, which means only the start urls will be crawled. To change that, you can specify via --max-depth . Or in a simpler way, use -m 2 to run the crawler in thorough mode which is equivalent to --max-depth 2 . By default the normal mode -m 1 is adop","default_branch":null,"files":null,"tree":[],"storefront":"/r/PadishahIII","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/PadishahIII/SecretScraper/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}