{"repo":"hominee/dyer","free":true,"listed":false,"github":"https://github.com/hominee/dyer","clone":"git clone https://github.com/hominee/dyer.git","description":"Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.","language":"Rust","stars":126,"topics":["crawler","web-crawler","web-scraping","web-framework","rust","spider","rust-programming-language"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"dyer Overview [dyer] is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, interoperable, comprehensive features without compromising speed. Features asynchronous, concurrent streaming and I/O, make the best of thread pool, network, and system resource. Event-driven, once you set the initials and recursive generator, dyer will handle the rest of it interoperably. User-friendly and flexible, dyer offers high-level, flexible, easy to use wrappers and APIs what does a lot for you. Feature Flag To reduce code redundancy and speed up compilation, dyer use feature flag to mark the necessary modules/functions, Currently here are some supported Features: - xpath-stable : stably parse the response html with third-party C library libxml2 - xpath-alpha : rust-native parse the html response, NOTE that it is experimental and unstale, remains to be improved - compression : Enable HTTP Compression: br , deflate , gzip - proxy : Enable use proxies - full : Enable all features Guide Get started by installing [dyer-cli] and looking over the [examples] and [quick start]. Crates: Link 🔗 Documentation: Link 🔗 The Cookbook gives a detailed view of dyer. [dyer]: https://docs.rs/dyer [examples]: https://github.com/hominee/dyer/tree/master/examples/ [quick start]: https://github.com/hominee/dyer/tree/master/quick-start.md/ [dyer-cli]: https://github.com/hominee/dyer-cli","default_branch":null,"files":null,"tree":[],"storefront":"/r/hominee","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/hominee/dyer/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}