{"repo":"sjdirect/abot","free":true,"listed":false,"github":"https://github.com/sjdirect/abot","clone":"git clone https://github.com/sjdirect/abot.git","description":"Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.","language":"C#","stars":2310,"topics":["c-sharp","abot","abot-nuget","crawler","web-crawler","parsing","spider","spiders","pluggable","unit-testing"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Abot Please star this project!! C# web crawler built for speed and flexibility. Abot is an open source C# web crawler framework built for speed and flexibility. It takes care of the low level plumbing (multithreading, http requests, scheduling, link parsing, etc..). You just register for events to process the page data. You can also plugin your own implementations of core interfaces to take complete control over the crawl process. Abot Nuget package version = 2.0 targets Dotnet Standard 2.0 and Abot Nuget package version Quick Start Installing Abot Install Abot using Nuget Using Abot Abot Configuration Abot's Abot2.Poco.CrawlConfiguration class has a ton of configuration options. You can see what effect each config value has on the crawl by looking at the code comments . Abot Events Register for events and create processing methods Custom objects and the dynamic crawl bag Add any number of custom objects to the dynamic crawl bag or page bag. These objects will be available in the CrawlContext.CrawlBag object, PageToCrawl.PageBag object or CrawledPage.PageBag object. Cancellation Customizing Crawl Behavior Abot was designed to be as pluggable as possible. This allows you to easily alter the way it works to suite your needs. The easiest way to change Abot's behavior for common features is to change the config values that control them. See the Quick Start page for examples on the different ways Abot can be configured. CrawlDecision Callbacks/Delegates Sometimes you don't want to","default_branch":null,"files":null,"tree":[],"storefront":"/r/sjdirect","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/sjdirect/abot/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}