{"repo":"kayslay/web-crawljs","free":true,"listed":false,"github":"https://github.com/kayslay/web-crawljs","clone":"git clone https://github.com/kayslay/web-crawljs.git","description":"web crawler for Nodejs","language":"JavaScript","stars":21,"topics":["web","crawler","nodejs","node","scrap","web-crawljs","cheerio","javascript"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"web-crawlerjs About web-crawlerjs is a package that extracts information from a web page. This package depends on request and cheerio. web-crawlerjs crawl pages using a Breadth-First Algorithm. Installation run this on your terminal or command prompt. $ npm install web-crawljs --save Example An example of the usage of web-crawljs. This example extracts data from https://example.com Understanding the example require('web-crawljs'); returns a factory function that takes in one argument. This argument is the configuration object. The configuration Object is used to configure the behaviour of crawler. const crawler = require('web-crawljs'); const config = {...} const Crawler = crawler(config) calling crawler(config) , in the example above returns a new Object, this object has a single method, CrawlAllUrl . CrawlAllUrl starts the crawl and returns a Promise when its done. The Promise success value or failure error depends on the finalFn function. More will be explained in the finalFn section. The config object The config object is the only argument passed into the crawler factory function. It configures the way the crawler would behave. This object configures the callbacks, the schemas of the selector, depth, the urls to visit e.t.c. Fetching data and setting next links to visit These properties are responsible for fetching data and getting the next links to visit next. The fetchSelector and nextSelector are used to assign the elements to scrap. The fetchSelectBy and nextSelectBy ","default_branch":null,"files":null,"tree":[],"storefront":"/r/kayslay","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/kayslay/web-crawljs/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}