{"repo":"cisnlp/GlotCC","free":true,"listed":false,"github":"https://github.com/cisnlp/GlotCC","clone":"git clone https://github.com/cisnlp/GlotCC.git","description":"[NeurIPS 2024] 🕸 GlotCC Dataset and Pipline","language":"Jupyter Notebook","stars":21,"topics":["glotcc","common-crawl","commoncrawl","corpus-linguistics","crawler","glot","language-identification","multlingual","multilingual-dataset","glotlid"],"license":"CC0-1.0","category":"scrapers-browser-automation","readme_excerpt":"GlotCC HomePage GlotCC is a multilingual corpus built by the GlotLID language identification and cisnlp/Ungoliant pipeline from CommonCrawl. Lastest version supports more than 1000 languages and is filtered based on adopted filters from C4, CCNet, MADLAD-400, RedPajama-Data-v2, OSCAR, Gopher, RefinedWeb, FineWeb, Datatrove, Dolma, Pile-CC, Pretrainer's Guide, and GlotScript. ™ The logo features a llama with the style of C.C. from the Code Geass anime reading a book. Dataset GlotCC Dataset, Version 1: https://huggingface.co/datasets/cis-lmu/GlotCC-V1 Running the pipeline We forked oscar-project/ungoliant to cisnlp/ungoliant and made the necessary changes to integrate it with the GlotLID language identification model. For detailed instructions on running the pipeline, refer to the cisnlp/ungoliant repository. The README is up-to-date. Acknowledgements - We appreciate the collaborators who are collectively advancing the frontier of open datasets and LLM models. - Thanks to the community and friends who enable the auditing of this dataset with higher quality. Also, to everyone contributing to the GlotCC dataset. - Our gratitude extends to the OSCAR team for pioneering the development of open pipelines and datasets from CommonCrawl, as well as to the CommonCrawl team. License - GlotCC data is released under the following licensing scheme: We do not own any of the text from which this data has been extracted. The data is licensed under the terms of the CommonCrawl Terms of Use. We ","default_branch":null,"files":null,"tree":[],"storefront":"/r/cisnlp","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/cisnlp/GlotCC/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}