{"repo":"internetarchive/Zeno","free":true,"listed":false,"github":"https://github.com/internetarchive/Zeno","clone":"git clone https://github.com/internetarchive/Zeno.git","description":"State-of-the-art web crawler 🔱","language":"Go","stars":421,"topics":["web-crawler","zeno","archiving"],"license":"AGPL-3.0","category":"scrapers-browser-automation","readme_excerpt":"Zeno State-of-the-art web crawler 🔱 Introduction Zeno is a web crawler designed to operate wide crawls or to simply archive one web page. Zeno's key concepts are: portability, performance, simplicity. With an emphasis on performance. It heavily relies on the gowarc module for traffic recording into WARC files. The name Zeno comes from Zenodotus (Ζηνόδοτος), a Greek grammarian, literary critic, Homeric scholar, and the first librarian of the Library of Alexandria. Requirements for Building - Go 1.25+ - As specified in go.mod - If CGO ENABLED=1 (enabled by default): GCC 12+ - Required for building C++ dependencies with C++20 constexpr support for the WHATWG URL parser (github.com/ada-url/goada). - If CGO ENABLED=0: No additional requirements, as the CGO-free WebAssembly wrapper of goada (goada-wasm) will be used. (1x slower than CGO version on amd64 and arm64, and 10x or more slower on other CPU architectures! Check https://wazero.io/docs/#compiler for details) Note: GCC 11 and earlier versions do not support the C++20 constexpr features required by the ada-url/goada dependency. On Ubuntu 22 LTS and earlier, you may need to install a newer GCC version or disable CGO. Installation or utilize our pre-built release binaries here, but do note that we are mainly focused on linux/amd64 support at this time. Quick Start To archive a single web page: Zeno is highly configurable with many parameters that can be customized. To see all available configuration options, use Zeno -h and/or ","default_branch":null,"files":null,"tree":[],"storefront":"/r/internetarchive","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/internetarchive/Zeno/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}