{"repo":"nvk681/Gumo","free":true,"listed":false,"github":"https://github.com/nvk681/Gumo","clone":"git clone https://github.com/nvk681/Gumo.git","description":"A crawler that extracts data from a dynamic webpage. Written in node js.","language":"JavaScript","stars":20,"topics":["elasticsearch","nodejs","crawler","neo4j"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"πŸ•ΈοΈGumo \"Gumo\" (θœ˜θ››) is Japanese for \"spider\". Overview πŸ‘“ A web-crawler (get it?) and scraper that extracts data from a family of nested dynamic webpages with added enhancements to assist in knowledge mining applications. Written in NodeJS. Table of Contents πŸ“– - πŸ•ΈοΈGumo - Overview πŸ‘“ - Table of Contents πŸ“– - Features 🌟 - Requirements πŸ“‹ - Installation πŸ—οΈ - Usage πŸ‘¨β€πŸ’» - Development πŸ› οΈ - Configuration βš™οΈ - ElasticSearch ⚑ - GraphDB β˜‹ - Nodes - Relationships - Changelog - TODO β˜‘οΈ Features 🌟 - Crawl hyperlinks present on the pages of any domain and its subdomains. - Scrape meta-tags and body text from every page. - Store entire sitemap in a GraphDB (currently supports Neo4J). - Store page content in ElasticSearch for easy full-text lookup. Requirements πŸ“‹ - Node.js β‰₯ 24.0.0 (LTS). Pinned in package.json ( engines ) and .nvmrc for nvm users. - Neo4j 4.0+ when using the graph (constraint syntax requires it). Installation πŸ—οΈ 1. Use Node 24+ (e.g. nvm use if you have nvm and the repo’s .nvmrc ). 2. Install dependencies (uses package-lock.json for reproducible installs): Or in CI: npm ci . Usage πŸ‘¨β€πŸ’» From code: Note: The config params passed to cron.configure above are the default values. See Configuration for all options. When using Gumo as a dependency (e.g. require('gumo') with no config.json in your project), in-package defaults are used so the module loads; pass your Elasticsearch, Neo4j, and crawler settings via configure() before calling insert() . Development πŸ› οΈ Scrip","default_branch":null,"files":null,"tree":[],"storefront":"/r/nvk681","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/nvk681/Gumo/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}