{"repo":"microlinkhq/cloudflare-bot-directory","free":true,"listed":false,"github":"https://github.com/microlinkhq/cloudflare-bot-directory","clone":"git clone https://github.com/microlinkhq/cloudflare-bot-directory.git","description":"Cloudflare Radar verified bots directory — 500+ crawlers and search-engine user agents as data.","language":"JavaScript","stars":16,"topics":["bots","cloudflare","crawlers","dataset","googlebot","user-agent","bot-list","directory","radar","verified"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"CloudFlare Bot Directory A list of 500+ verified bots and web crawlers sourced from CloudFlare Radar, available as a JSON dataset for bot detection, user agent analysis, and web scraping identification. [!NOTE] Not affiliated with or endorsed by Cloudflare, Inc. The dataset is licensed CC BY-NC 4.0 and limited to non-commercial use — see License. Why Identifying legitimate bots from malicious scrapers is essential for web security and analytics. This package repackages the CloudFlare Radar verified bots directory as data, helping you: - Detect verified bots – Identify legitimate crawlers like Googlebot, Bingbot, and more - Filter analytics – Exclude known bots from your traffic reports - Allow-list crawlers – Permit verified bots while blocking suspicious traffic - User agent lookup – Match user agent strings against known bot patterns Data Structure Each bot entry includes: Field Description ------------------- ---------------------------------------------- slug URL-friendly unique identifier name Human-readable bot name kind Bot kind classification operator Company or organization operating the bot operatorUrl URL to operator's documentation category Bot category (Search Engine, Monitoring, etc.) description What the bot does followsRobotsTxt Whether the bot respects robots.txt userAgentPatterns User agent pattern(s) for matching userAgents Known user agent string(s) Install Usage Related - top-user-agents – An always up-to-date list of the top 100 HTTP user-agents most use","default_branch":null,"files":null,"tree":[],"storefront":"/r/microlinkhq","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/microlinkhq/cloudflare-bot-directory/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}