{"repo":"oscar-project/ungoliant","free":true,"listed":false,"github":"https://github.com/oscar-project/ungoliant","clone":"git clone https://github.com/oscar-project/ungoliant.git","description":":spider: The pipeline for the OSCAR corpus","language":"Rust","stars":178,"topics":["commoncrawl","oscar","crawler","language-classification","corpus-linguistics","nlp","fasttext","common-crawl"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Ungoliant 🕷️ Ungoliant is a high-performance pipeline that provides tools to build corpus generation pipelines from CommonCrawl. 🕷️ It currently is the generation pipeline for OSCAR corpus, from CommonCrawl. Ungoliant is a replacement of goclassy. Installation Installing/Compiling the binary Via cargo : cargo install ungoliant Via git : cargo install --git https://github.com/oscar-corpus/ungoliant Ungoliant needs numerous dependencies that should be compiled when installing. However cmake / gcc can be needed as the project uses fasttext-rs. KenLM feature The KenLM feature is optional because it relies on unsafe code that can break if the supplied model files are not correct. To enable it, install KenLM requirements: and use cargo install ungoliant --features kenlm or cargo b --features kenlm if you're building from source. Getting a language identification file (for fastText): By default, ungoliant expects the lid.176.bin model by meta. Use curl https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin -o lid.176.bin to get it. However, you can use the model you want: just point to its path using ungoliant download --lid-path . Other options include: - NLLB model (https://huggingface.co/facebook/fasttext-language-identification) - OpenLID model (https://github.com/laurieburchell/open-lid-dataset) Usage The usual way of generating corpora is: 1. Fetch the wet.paths.gz file from the last CommonCrawl dump and decompress it. 2. Download the files using the download ","default_branch":null,"files":null,"tree":[],"storefront":"/r/oscar-project","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/oscar-project/ungoliant/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}