{"repo":"bda-research/node-crawler","free":true,"listed":false,"github":"https://github.com/bda-research/node-crawler","clone":"git clone https://github.com/bda-research/node-crawler.git","description":"Web Crawler/Spider for NodeJS + server-side jQuery ;-)","language":"TypeScript","stars":6798,"topics":["crawler","javascript","spider","extract-data","cheerio","jquery","nodejs"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"------ [![NPM download][download-image]][download-url] [![Package Quality][quality-image]][quality-url] [quality-image]: https://packagequality.com/shield/crawler.svg [quality-url]: https://packagequality.com/#?package=crawler [download-image]: https://img.shields.io/npm/dm/crawler.svg?style=flat-square [download-url]: https://npmjs.org/package/crawler Crawler v2 : Advanced and Typescript version of node-crawler Features: - Server-side DOM & automatic jQuery insertion with Cheerio (default), - Configurable pool size and retries, - Control rate limit, - Priority queue of requests, - let crawler deal for you with charset detection and conversion, If you have prior experience with Crawler v1, for fast migration, please proceed to the section Differences and Breaking Changes. Use with AI agents This package ships an agent skill that teaches AI agents (e.g. OpenClaw, Hermes, Claude Code, Codex etc) how to crawl with crawler — queue vs. direct requests, rate limiting, proxy rotation, Cheerio parsing, and common pitfalls. Recommended — install from ClawHub: clawhub.ai/mike442144/node-crawler Fallback — download the skill bundle from the latest release and unzip it into your agent's skills directory: Let your agent install it — paste this prompt: Install the node-crawler skill so you can use it for large-scale web crawling: download https://github.com/bda-research/node-crawler/releases/latest/download/node-crawler-skill.zip , unzip it, and place the node-crawler/ folder in your skill","default_branch":null,"files":null,"tree":[],"storefront":"/r/bda-research","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/bda-research/node-crawler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}