{"repo":"JaCraig/Spidey","free":true,"listed":false,"github":"https://github.com/JaCraig/Spidey","clone":"git clone https://github.com/JaCraig/Spidey.git","description":"A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.","language":"C#","stars":16,"topics":["crawler","webcrawler"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Spidey Spidey is a flexible and extensible .NET library for crawling web content. It is designed for .NET Core applications and provides a modular architecture, allowing you to customize or extend any part of the crawling pipeline. Features - Simple API for crawling websites - Highly configurable via the Options class - Dependency injection support (IoC/DI) - Easily replaceable subsystems (engine, parser, scheduler, etc.) - Callback-based result handling - NuGet package available Quick Start Install the NuGet package: Setting up the Library Register Spidey in your app's service collection using the RegisterSpidey extension method: Alternatively, you can instantiate Crawler and Options directly without DI: Options Configuration The Options class configures the crawler's behavior. Key properties include: - ItemFound ( Action ): Callback invoked when a new page is discovered. - Allow ( List ): Regex patterns for URLs allowed to be crawled. - FollowOnly ( List ): Regex patterns for pages whose links should be followed. - Ignore ( List ): Regex patterns for URLs to ignore. - StartLocations ( List ): Initial URLs to start crawling from. - UrlReplacements ( Dictionary ): URL replacements during crawling. - NetworkCredentials ( NetworkCredential ): Optional credentials for authentication. - UseDefaultCredentials ( bool ): Use default system credentials. - Proxy ( IWebProxy ): Optional proxy settings. Example callback method: Basic Usage Once configured, start the crawl process: The l","default_branch":null,"files":null,"tree":[],"storefront":"/r/JaCraig","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/JaCraig/Spidey/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}