{"repo":"spatie/robots-txt","free":true,"listed":false,"github":"https://github.com/spatie/robots-txt","clone":"git clone https://github.com/spatie/robots-txt.git","description":"Determine if a page may be crawled from robots.txt, robots meta tags and robot headers","language":"PHP","stars":258,"topics":["php","robots-txt","crawler"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Parse robots.txt , robots meta and headers Determine if a page may be crawled from robots.txt, robots meta tags and robot headers. Support us We invest a lot of resources into creating best in class open source packages. You can support us by buying one of our paid products. We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on our contact page. We publish all received postcards on our virtual postcard wall. Installation You can install the package via composer: Usage You can also specify a user agent: By default, Robots will look for a robots.txt file on https://host.com/robots.txt . Another location can be specified like so: Testing Changelog Please see CHANGELOG for more information what has changed recently. Contributing Please see CONTRIBUTING for details. Security Vulnerabilities Please review our security policy on how to report security vulnerabilities. Postcardware You're free to use this package, but if it makes it to your production environment we highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. Our address is: Spatie, Kruikstraat 22, 2018 Antwerp, Belgium. We publish all received postcards on our company website. Credits - Brent Roose - All Contributors License The MIT License (MIT). Please see License File for more information.","default_branch":null,"files":null,"tree":[],"storefront":"/r/spatie","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/spatie/robots-txt/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}