{"repo":"duckduckgo/tracker-radar-collector","free":true,"listed":false,"github":"https://github.com/duckduckgo/tracker-radar-collector","clone":"git clone https://github.com/duckduckgo/tracker-radar-collector.git","description":"🕸 Modular, multithreaded, puppeteer-based crawler","language":"JavaScript","stars":166,"topics":["tracker-radar","crawler","puppeteer"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"DuckDuckGo Tracker Radar Collector 🕸 Modular, multithreaded, puppeteer-based crawler used to generate third party request data for the Tracker Radar. How do I use it? Use it from the command line 1. Clone this project locally ( git clone git@github.com:duckduckgo/tracker-radar-collector.git ) 2. Install all dependencies ( npm i ) 3. Run the command line tool: Available options: - -o, --output - (required) output folder where output files will be created - -u, --url - single URL to crawl - -i, --input-list - path to a text file with list of URLs to crawl (each in a separate line) - -d, --data-collectors - comma separated list (e.g -d 'requests,cookies' ) of data collectors that should be used (all by default) - -c, --crawlers - override the default number of concurrent crawlers (default number is picked based on the number of CPU cores) - --reporters - comma separated list (e.g. --reporters 'cli,file,html' ) of reporters to be used ('cli' by default) - -v, --verbose - instructs reporters to log additional information (e.g. for \"cli\" reporter progress bar will not be shown when verbose logging is enabled) - -l, --log-path - instructs reporters where all logs should be written to - -f, --force-overwrite - overwrite existing output files (by default entries with existing output files are skipped) - -3, --only-3p - don't save any first-party data (e.g. requests, API calls for the same eTLD+1 as the main document) - -m, --mobile - emulate a mobile device when crawling - -p, --prox","default_branch":null,"files":null,"tree":[],"storefront":"/r/duckduckgo","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/duckduckgo/tracker-radar-collector/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}