{"repo":"elixir-crawly/crawly","free":true,"listed":false,"github":"https://github.com/elixir-crawly/crawly","clone":"git clone https://github.com/elixir-crawly/crawly.git","description":"Crawly, a high-level web crawling & scraping framework for Elixir.","language":"Elixir","stars":1114,"topics":["elixir","erlang","scraper","scraping","scraping-websites","extract-data","spider","crawler","crawling"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Crawly Overview Crawly is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival. Requirements 1. Elixir 1.14 2. Works on GNU/Linux, Windows, macOS X, and BSD. Quickstart 0. Create a new project: mix new quickstart --sup 1. Add Crawly as a dependencies: 2. Fetch dependencies: $ mix deps.get 3. Create a spider New in 0.15.0 : It's possible to use the command to speed up the spider creation, so you will have a generated file with all needed callbacks: mix crawly.gen.spider --filepath ./lib/crawly example/books to scrape.ex --spidername BooksToScrape 4. Configure Crawly By default, Crawly does not require any configuration. But obviously you will need a configuration for fine tuning the crawls: (in file: config/config.exs ) New in 0.15.0: You can generate example config with the help of the following command: mix crawly.gen.config 5. Start the Crawl: 6. Results can be seen with: Running Crawly without Elixir or Elixir projects It's possible to run Crawly in a standalone mode, when Crawly is running as a tiny docker container, and spiders are just YMLfiles or elixir modules that are mounted inside. Please read more about it here: - Standalone Crawly - Spiders as YML Need more help? Please use discussions for all conversations related to the project Browser rendering Crawly can be configured in the way that all fetched pages will be bro","default_branch":null,"files":null,"tree":[],"storefront":"/r/elixir-crawly","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/elixir-crawly/crawly/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}