{"repo":"loadkpi/crawler_detect","free":true,"listed":false,"github":"https://github.com/loadkpi/crawler_detect","clone":"git clone https://github.com/loadkpi/crawler_detect.git","description":"Ruby gem to detect bots and crawlers via the user agent","language":"Ruby","stars":148,"topics":["ruby","crawler","spider","bots","crawler-detection","bot-detection"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"CrawlerDetect About CrawlerDetect is a Ruby version of PHP class @CrawlerDetect. It helps to detect bots/crawlers/spiders via the user agent and other HTTP-headers. Currently able to detect 1,000's of bots/spiders/crawlers. Why CrawlerDetect? Comparing with other popular bot-detection gems: CrawlerDetect Voight-Kampff Browser -- -- -- -- Number of bot-patterns 1000 280 280 Number of checked HTTP-headers 11 1 1 Number of updates of bot-list (1st half of 2018) 14 1 7 In order to remain up-to-date, this gem does not accept any crawler data updates – any PRs to edit the crawler data should be offered to the original JayBizzle/CrawlerDetect project. Requirements - Ruby: MRI 2.5+ or JRuby 9.3+. Installation Add this line to your application's Gemfile: gem 'crawler detect' Basic Usage Or if you need crawler name: Rack::Request extension Optionally you can add additional methods for request : It's more flexible to use request.is crawler? rather than CrawlerDetect.is crawler? because it automatically checks 10 HTTP-headers, not only HTTP USER AGENT . Only one thing you have to do is to configure Rack::CrawlerDetect middleware: Rails Rack Configuration In some cases you may want to use your own white-list, or black-list or list of http-headers to detect User-agent. It is possible to do via CrawlerDetect::Config . For example, you may have initializer like this: Make sure that your files are correct JSON files. Look at the raw files which are used by default for more information. Develo","default_branch":null,"files":null,"tree":[],"storefront":"/r/loadkpi","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/loadkpi/crawler_detect/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}