{"repo":"bkeepers/spiderman","free":true,"listed":false,"github":"https://github.com/bkeepers/spiderman","clone":"git clone https://github.com/bkeepers/spiderman.git","description":"your friendly neighborhood web crawler","language":"Ruby","stars":18,"topics":["webcrawler","crawler","crawler-engine","spider","web-scraping","ruby","spider-framework","nokogiri","http","httprb"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"your friendly neighborhood web crawler Spiderman is a Ruby gem for crawling and processing web pages. Installation Add this line to your application's Gemfile: And then execute: $ bundle install Or install it yourself as: $ gem install spiderman Usage Run the crawler: ActiveJob Spiderman works with ActiveJob out of the box. If your crawler class inherits from ActiveJob:Base , then requests will be made in your background worker. Each request will run as a separate job. Development After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment. To install this gem onto your local machine, run bundle exec rake install . To release a new version, update the version number in version.rb , and then run bundle exec rake release , which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org. Contributing Bug reports and pull requests are welcome on GitHub at https://github.com/bkeepers/spiderman. License The gem is available as open source under the terms of the MIT License.","default_branch":null,"files":null,"tree":[],"storefront":"/r/bkeepers","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/bkeepers/spiderman/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}