{"repo":"yields/ant","free":true,"listed":false,"github":"https://github.com/yields/ant","clone":"git clone https://github.com/yields/ant.git","description":"A web crawler for Go","language":"Go","stars":281,"topics":["web-crawler","golang","scraper","go","spider"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"ant ( alpha ) is a web crawler for Go. Declarative The package includes functions that can scan data from the page into your structs or slice of structs, this allows you to reduce the noise and complexity in your source-code. You can also use a jQuery-like API that allows you to scrape complex HTML pages if needed. Headless By default the crawler uses http.Client , however if you're crawling SPAs youc an use the antcdp.Client implementation which allows you to use chrome headless browser to crawl pages. Polite The crawler automatically fetches and caches robots.txt , making sure that it never causes issues to small website owners. Of-course you can disable this behavior. Concurrent The crawler maintains a configurable amount of \"worker\" goroutines that read URLs off the queue, and spawn a goroutine for each URL. Depending on your configuration, you may want to increase the number of workers to speed up URL reads, of-course if you don't have enough resources you can reduce the number of workers too. Rate limits The package includes a powerful ant.Limiter interface that allows you to define rate limits per URL. There are some built-in limiters as well. Note that LimitPattern and LimitRegexp only match on the host and path of the URL. Matchers Another powerful interface is ant.Matcher which allows you to define URL matchers, the matchers are called before URLs are queued. Robust The crawl engine automatically retries any errors that implement Temporary() error that returns true.","default_branch":null,"files":null,"tree":[],"storefront":"/r/yields","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/yields/ant/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}