{"repo":"cxcscmu/Craw4LLM","free":true,"listed":false,"github":"https://github.com/cxcscmu/Craw4LLM","clone":"git clone https://github.com/cxcscmu/Craw4LLM.git","description":"Official repository for \"Craw4LLM: Efficient Web Crawling for LLM Pretraining\"","language":"Python","stars":663,"topics":["crawler","crawling","large-language-models","llm","pre-training","pretraining","web-crawler","web-crawling"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Craw4LLM This repo contains the code for the paper \"Craw4LLM: Efficient Web Crawling for LLM Pretraining\". Prerequisite 1. Request the ClueWeb22 dataset. 2. Create a virtual environment with python = 3.10 and install the following requirements: 3. Download the DCLM fastText classifier to fasttext scorers/ . [!IMPORTANT] To run the crawler efficiently, the ClueWeb22 data should be placed on an SSD . Run the Crawler To run a (simulated) crawl, first create a yaml configuration file under configs/ , and run the following command: Craw4LLM Create a yaml file in configs/ with the following content: Documents are scored by all scorers in rating methods . In the above configuration file, we set a length scorer, which scores a document by its length, and a fasttext score scorer which uses the DCLM fastText model to score a document. The final ranking is determined by selection method which is set to dclm fasttext score , the name of the fasttext score scorer. Baseline Crawlers Random Crawler Indegree-based Crawler Pretraining and Evaluation After running the crawler, the crawled document ids will be placed in output dir in the configuration file. Run the following command to get the document texts: Then you can use the DCLM framework to run LLM pretraining and evaluation. Miscellaneous Browse the Data Run the following command to print a document and its outlinks by its id:","default_branch":null,"files":null,"tree":[],"storefront":"/r/cxcscmu","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/cxcscmu/Craw4LLM/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}