{"repo":"gocolly/colly","free":true,"listed":false,"github":"https://github.com/gocolly/colly","clone":"git clone https://github.com/gocolly/colly.git","description":"Elegant Scraper and Crawler Framework for Golang","language":"Go","stars":25428,"topics":["golang","scraper","framework","crawler","scraping","crawling","spider","go"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Colly Lightning Fast and Elegant Scraping Framework for Gophers Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving. Sponsor NodeMaven: The most efficient proxy provider for Web Scrapping and Automation with the Highest Quality IP on the market. Why NodeMaven? - 99.9% uptime - ZIP Targeting - IP filtering: all proxies have fraud score 1k request/sec on a single core) - Manages request delays and maximum concurrency per domain - Automatic cookie and session handling - Sync/async/parallel scraping - Caching - Automatic encoding of non-unicode responses - Robots.txt support - Distributed scraping - Configuration via environment variables - Extensions Example See examples folder for more detailed examples. Installation go get github.com/gocolly/colly/v2 Bugs Bugs or suggestions? Visit the issue tracker or join #colly on freenode Other Projects Using Colly Below is a list of public, open source projects that use Colly: - greenpeace/check-my-pages Scraping script to test the Spanish Greenpeace web archive. - altsab/gowap Wappalyzer implementation in Go. - jesuiscamille/goquotes A quotes scraper, making your day a little better! - jivesearch/jivesearch A search engine that doesn't track you. - Leagify/colly-draft-prospects A scraper for future NFL Draft prospects. - lucasepe/go-ps4 Search playsta","default_branch":null,"files":null,"tree":[],"storefront":"/r/gocolly","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/gocolly/colly/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}