{"repo":"mindfiredigital/DeepScanBot","free":true,"listed":false,"github":"https://github.com/mindfiredigital/DeepScanBot","clone":"git clone https://github.com/mindfiredigital/DeepScanBot.git","description":"It allows you to crawl websites with various configurations, including crawl depth, timeout settings, proxy support, and output options.","language":"Go","stars":11,"topics":["bot","crawl","crawler","go","golang","google","webcrawler"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"DeepScanBot CLI A high-performance web crawler and scanner — recursively crawls websites, respects robots.txt, handles rate-limiting, and produces comprehensive JSON or text reports. Available as a single, self-contained binary via npm. What is DeepScanBot? DeepScanBot is a feature-rich, concurrent web crawler that makes website crawling fast, reliable, and developer-friendly. It handles the complexities of web crawling automatically: - Discovers all pages on a website by following internal links up to a configurable depth - Respects crawl rules defined in robots.txt with an option to bypass when needed - Handles rate limits with automatic retry and exponential backoff - Filters content by MIME type for targeted crawling - Exports results in structured JSON or readable text formats with detailed analytics Who Is It For? - Security Researchers - Audit websites for exposed endpoints - SEO Specialists - Discover all pages, find broken links, analyze site structure - QA Engineers - Verify page availability, test website coverage - Data Analysts - Collect structured data from websites for analysis - DevOps Engineers - Monitor website health, validate deployments Features Here's what DeepScanBot can do for you: - Concurrent Crawling : Multi-threaded architecture with configurable concurrency, per-host rate limiting, and CPU-aware auto-scaling - Robots.txt Compliance : Automatically respects robots.txt rules with optional bypass - Retry & Rate-Limit Handling : Automatic retry with e","default_branch":null,"files":null,"tree":[],"storefront":"/r/mindfiredigital","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mindfiredigital/DeepScanBot/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}