{"repo":"myvyang/chromium_for_spider","free":true,"listed":false,"github":"https://github.com/myvyang/chromium_for_spider","clone":"git clone https://github.com/myvyang/chromium_for_spider.git","description":"dynamic crawler for web vulnerability scanner","language":"HTML","stars":250,"topics":["security","crawler","chromium","puppeteer","spider"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"中文版本(Chinese version) Core functions By modifying the chromium source code, two functions that are more concerned about in two dynamic crawlers are mainly implemented: 1. The current page is prohibited from being redirected, and the URL to be redirected is collected for future use. This is achieved by modifying a relatively low-level function, and there is no need to hook in various scenarios. 2. Hook all non-default events bound by the current page and reserve the scene for subsequent triggers. This way the crawler does not need to traverse all DOM nodes. There are also some small features: 1. Disable browsers from downloading files. 2. Ignore the \"X-Frame-Options\" header, which allows arbitrary pages to be iframed. 3. Prohibit alert, print, confirm, prompt popups. 4. Forbid the page to open a new window by itself; record the URL of the new window to be opened. 5. Hidden the navigator.webdriver property. Download Compiled binary: https://github.com/myvyang/chromium for spider/releases Example After opening the page, because the page jump is hooked, no matter how you click (or execute any JS), the page should not jump successfully. \"URL to be redirected\" will be recorded in window.info . Instructions for use After implementing the core function by modifying the chromium source code, three properties have been added to the browser's window object: window.info , window.eventNames , and window.eventNodes . Among them, window.info records the jump URL triggered in the page, etc.,","default_branch":null,"files":null,"tree":[],"storefront":"/r/myvyang","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/myvyang/chromium_for_spider/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}