{"repo":"crwlrsoft/robots-txt","free":true,"listed":false,"github":"https://github.com/crwlrsoft/robots-txt","clone":"git clone https://github.com/crwlrsoft/robots-txt.git","description":"Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping","language":"PHP","stars":10,"topics":["robots-txt-parser","web-crawling","hacktoberfest","robots-txt","robots-exclusion-standard","robots-exclusion-protocol","web-scraping"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping Use this library within crawler/scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths. Documentation You can find the documentation at crwlr.software. Contributing If you consider contributing something to this package, read the contribution guide (CONTRIBUTING.md).","default_branch":null,"files":null,"tree":[],"storefront":"/r/crwlrsoft","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/crwlrsoft/robots-txt/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}