{"repo":"michaeltelford/wgit","free":true,"listed":false,"github":"https://github.com/michaeltelford/wgit","clone":"git clone https://github.com/michaeltelford/wgit.git","description":"Wgit enables you to crawl and extract the data you want from the web","language":"Ruby","stars":16,"topics":["ruby","search-engine","web-crawler","mongodb","analytics","data-mining","web-scraper"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Wgit --- Wgit is a HTML web crawler, written in Ruby, that allows you to programmatically extract the data you want from the web. Wgit was primarily designed to crawl static HTML websites to index and search their content - providing the basis of any search engine; but Wgit is suitable for many application domains including: - URL parsing - Document content extraction (data mining etc) - Recursive website crawling (indexing, statistical analysis etc) Wgit provides a high level, easy-to-use API and DSL that you can use in your own applications and scripts. Check out this demo search engine - built using Wgit, Sinatra and MongoDB - deployed to fly.io. Try searching for something that's Ruby related like \"Matz\" or \"Rails\". Table Of Contents 1. Usage 2. Why Wgit? 3. Why Not Wgit? 4. Installation 5. Documentation 6. Executable 7. License 8. Contributing 9. Development Usage Let's crawl a quotes website extracting its quotes and authors using the Wgit DSL: Which outputs: Great! But what if we want to crawl and store the content in a database, so that it can be searched? Wgit makes it easy to index and search HTML using MongoDB (by default): The search call (on the last line) will return and output the results: Using a database client, we can see that the two web pages have been indexed, along with their extracted quotes and authors : The DSL makes it easy to write scripts for experimenting with. Wgit's DSL is simply a wrapper around the underlying classes. For comparison, here is t","default_branch":null,"files":null,"tree":[],"storefront":"/r/michaeltelford","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/michaeltelford/wgit/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}