{"repo":"Qianlitp/crawlergo","free":true,"listed":false,"github":"https://github.com/Qianlitp/crawlergo","clone":"git clone https://github.com/Qianlitp/crawlergo.git","description":"A powerful browser crawler for web vulnerability scanners","language":"Go","stars":3035,"topics":["headless-chrome","crawler","chrome-devtools","golang","vulnerability-scanner","headless","web-vulnerability-scanners","chromedp","crawlergo","blackhat"],"license":"GPL-3.0","category":"scrapers-browser-automation","readme_excerpt":"crawlergo A powerful browser crawler for web vulnerability scanners English Document 中文文档 crawlergo is a browser crawler that uses chrome headless mode for URL collection. It hooks key positions of the whole web page with DOM rendering stage, automatically fills and submits forms, with intelligent JS event triggering, and collects as many entries exposed by the website as possible. The built-in URL de-duplication module filters out a large number of pseudo-static URLs, still maintains a fast parsing and crawling speed for large websites, and finally gets a high-quality collection of request results. crawlergo currently supports the following features: chrome browser environment rendering Intelligent form filling, automated submission Full DOM event collection with automated triggering Smart URL de-duplication to remove most duplicate requests Intelligent analysis of web pages and collection of URLs, including javascript file content, page comments, robots.txt files and automatic Fuzz of common paths Support Host binding, automatically fix and add Referer Support browser request proxy Support pushing the results to passive web vulnerability scanners Screenshot Installation Please read and confirm disclaimer carefully before installing and using。 Build - compilation for current platform - compile for all platforms 1. crawlergo relies only on the chrome environment to run, go to download for the new version of chromium. 2. Go to download page for the latest version of crawlergo ","default_branch":null,"files":null,"tree":[],"storefront":"/r/Qianlitp","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Qianlitp/crawlergo/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}