{"repo":"data-prep-kit/data-prep-kit","free":true,"listed":false,"github":"https://github.com/data-prep-kit/data-prep-kit","clone":"git clone https://github.com/data-prep-kit/data-prep-kit.git","description":"Open source project for data preparation for GenAI applications","language":"HTML","stars":954,"topics":["data-preparation","finetuning","llm","llmapps","data","data-prep","data-preprocessing","data-preprocessing-pipelines","datacuration","large-language-models"],"license":"Apache-2.0","category":"ai-agents","readme_excerpt":"Data Prep Kit accelerates unstructured data preparation for LLM app developers. Developers can use Data Prep Kit to cleanse, transform, and enrich use case-specific unstructured data to pre-train LLMs, fine-tune LLMs, instruct-tune LLMs, or build Retrieval Augmented Generation (RAG) applications for LLMs Data Prep Kit can readily scale from a commodity laptop all the way to data center scale. Features - The kit provides a growing set of modules/transforms targeting laptop-scale to datacenter-scale processing. - The data modalities supported today are: Natural Language, Code, and Image. - The modules are built on common frameworks for Python and Ray runtimes for scaling up data processing. - The kit provides a framework for developing custom transforms for processing Parquet files as well as ZIP, NDJSON, and JSONL file formats. - The kit provides examples of how a single transform can be deployed on Kubernetes clusters as a Python or a Ray job. Additionally, when multiple transforms are deployed in a sequence, the kit uses Tekton pipelines. Installation The latest version of the Data Prep Kit is available on PyPi for Python 3.10, 3.11, 3.12, and 3.13. It can be installed using: This will install all available transforms. For guidance on creating the virtual environment for installing the data prep kit, click here. 🚀 Getting Started Fastest way to experience Data Prep Kit With no setup necessary, let's use a Google Colab friendly notebook to try Data Prep Kit. This is a","default_branch":null,"files":null,"tree":[],"storefront":"/r/data-prep-kit","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/data-prep-kit/data-prep-kit/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}