{"repo":"datajuicer/data-juicer","free":true,"listed":false,"github":"https://github.com/datajuicer/data-juicer","clone":"git clone https://github.com/datajuicer/data-juicer.git","description":"Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷","language":"Python","stars":6897,"topics":["data-analysis","data-science","large-language-models","llm","data-visualization","llms","instruction-tuning","pre-training","multi-modal","synthetic-data"],"license":"Apache-2.0","category":"ai-agents","readme_excerpt":"Data-Juicer: The Data Operating System for the Foundation Model Era Multimodal Cloud-Native AI-Ready Large-Scale Data-Juicer (DJ) transforms raw data chaos into AI-ready intelligence. It treats data processing as composable infrastructure —providing modular building blocks to clean, synthesize, and analyze data across the entire AI lifecycle, unlocking latent value in every byte. Whether you're deduplicating web-scale pre-training corpora, curating agent interaction traces, or preparing domain-specific RAG indices, DJ scales seamlessly from your laptop to thousand-node clusters—no glue code required. Alibaba Cloud PAI has deeply integrated Data-Juicer into its data processing products. See Quickly submit a DataJuicer job . --- 🚀 Quick Start Zero-install exploration : - JupyterLab Playground with Tutorials - Ask DJ Copilot Install & run : Or compose in Python : --- ✨ Why Data-Juicer? 1. Modular & Extensible Architecture - 200+ operators spanning text, image, audio, video, and multimodal data - Recipe-first : Reproducible YAML pipelines you can version, share, and fork like code - Composable : Drop in a single operator, chain complex workflows, or orchestrate full pipelines - Hot-reload : Iterate on operators without pipeline restarts 2. Full-Spectrum Data Intelligence - Foundation Models : Pre-training, fine-tuning, RL, and evaluation-grade curation - Agent Systems : Clean tool traces, structure context, de-identification, and quality gating - RAG & Analytics : Extraction, no","default_branch":null,"files":null,"tree":[],"storefront":"/r/datajuicer","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/datajuicer/data-juicer/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}