{"repo":"seantomburke/sitemapper","free":true,"listed":false,"github":"https://github.com/seantomburke/sitemapper","clone":"git clone https://github.com/seantomburke/sitemapper.git","description":"Parse through any sitemap in Node.js","language":"TypeScript","stars":137,"topics":["sitemap","sitemap-xml","parse","javascript","crawler","crawling","indexing","robots-txt","seo","sitemapper"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"📋 Overview Sitemapper is a Node.js module that makes it easy to parse XML sitemaps. It supports single sitemaps, sitemap indexes with multiple sitemaps, and various sitemap formats including image and video sitemaps. 🚀 Installation 🏃‍♂️ Quick Start Module Usage CLI Usage You can also use Sitemapper directly from the command line: 💻 Examples Promise Example Async/Await Example Advanced Example with Proxy ⚙️ Configuration Options Sitemapper can be customized with the following options: Option Type Default Description url String undefined The URL of the sitemap to parse timeout Number 15000 Maximum timeout in milliseconds for each request concurrency Number 10 Maximum number of concurrent requests when crawling multiple sitemaps retries Number 0 Number of retry attempts for failed requests debug Boolean false Enable debug logging rejectUnauthorized Boolean true Reject invalid SSL certificates (like self-signed or expired) requestHeaders Object {} Additional HTTP headers to include with requests lastmod Number undefined Only return URLs with lastmod timestamp newer than this value proxyAgent HttpProxyAgent HttpsProxyAgent undefined Instance of hpagent for proxy support exclusions Array<RegExp> [] Array of regex patterns to exclude URLs from results fields Object undefined Specify which fields to include in the results (see below) Available Fields Important : When using the fields option, the return format changes from an array of URL strings to an array of objects conta","default_branch":null,"files":null,"tree":[],"storefront":"/r/seantomburke","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/seantomburke/sitemapper/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}