{"repo":"nadar/crawler","free":true,"listed":false,"github":"https://github.com/nadar/crawler","clone":"git clone https://github.com/nadar/crawler.git","description":"A Website Crawler Implementation written in PHP. High extendible, Indexes PDFs and is very memory efficient.","language":"PHP","stars":10,"topics":["php","crawler","html","pdf","hacktoberfest"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Website Crawler for PHP A highly extendible, dependency free Crawler for HTML, PDFS or any other type of Documents. Why another Page Crawler? Yes, indeed, there are already very good Crawlers around, therefore those where my goals: + Dependency Free - we don't want to use any HTTP client, as much \"native\" PHP code as possible in order to keep the overhead small. It just requires the CURL extension. + Memory Efficent - As memory efficient as possible, less overhead, full code control. + Extendible - Attach your own parsers in order to determine how html or any other format is parsed. There are out of the box parsers for HTML and PDF. Its very easy to build your own data type parser. + Runtime Storage - When the crawler runs, certain informations must be stored. This is extendible to suit your use case. Either use your database or take the built in array or file storage system. + Async - It's possible to start the crawler and process any further run cycle as an asynchronus process, f.e. with a PHP queue system like Yii2 Queue. Installation Composer is required to install this library: In order to use the PDF Parser, the optional library smalot/pdfparser must be installed: Usage 1. First we need to provide the crawler the information what should be done with the results from a crawler run: Create your handler, those are the classes which interact with the crawler in order to store your content/results somwehere. The afterRun() method will run whenever an URL is crawled and conta","default_branch":null,"files":null,"tree":[],"storefront":"/r/nadar","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/nadar/crawler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}