{"repo":"fredwu/crawler","free":true,"listed":false,"github":"https://github.com/fredwu/crawler","clone":"git clone https://github.com/fredwu/crawler.git","description":"A high performance web crawler / scraper in Elixir.","language":"Elixir","stars":956,"topics":["elixir","crawler","spider","scraper","scraper-engine","offline","files"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"Crawler A high performance web crawler / scraper in Elixir, with worker pooling and rate limiting via OPQ. Features - Crawl assets (javascript, css and images). - Save to disk. - Hook for scraping content. - Restrict crawlable domains, paths or content types. - Limit concurrent crawlers. - Limit rate of crawling. - Set the maximum crawl depth. - Set timeouts. - Set retries strategy. - Set crawler's user agent. - Manually pause/resume/stop the crawler. See Hex documentation. Architecture Below is a very high level architecture diagram demonstrating how Crawler works. Usage There are several ways to access the crawled page data: 1. Use Crawler.Store 2. Tap into the registry(?) Crawler.Store.DB 3. Use your own scraper 4. If the :save to option is set, pages will be saved to disk in addition to the above mentioned places 5. Provide your own custom parser and manage how data is stored and accessed yourself Configurations Option Type Default Value Description ------------- ------- --------------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- :assets list [] Whether to fetch any asset files, available options: \"css\" , \"js\" , \"images\" . :save to string nil When provided, the path for saving crawled pages. :workers integer 10 Maximum number of concurrent workers for crawling. :interval integer 0 Rate limit control - number of milliseco","default_branch":null,"files":null,"tree":[],"storefront":"/r/fredwu","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/fredwu/crawler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}