{"repo":"3DCF-Labs/doc2dataset","free":true,"listed":false,"github":"https://github.com/3DCF-Labs/doc2dataset","clone":"git clone https://github.com/3DCF-Labs/doc2dataset.git","description":"3DCF / doc2dataset: token-efficient document layer with NumGuard numeric integrity and multi-framework exports for RAG & fine-tuning.","language":"Rust","stars":55,"topics":["cli","data-pipeline","dataset-generation","document-processing","document-understanding","evaluation","fine-tuning","llm","machine-learning","nlp"],"license":"Apache-2.0","category":"machine-learning","readme_excerpt":"3DCF / doc2dataset Open document layer and doc→dataset pipeline for LLMs, with NumGuard numeric integrity and multi-framework exports. 3DCF/doc2dataset ingests PDFs, Markdown, plain text and other text-like formats into a normalized index ( documents.jsonl , pages.jsonl , cells.jsonl ), extracts NumGuard hashes for numeric cells, and generates QA/Summary/RAG datasets plus exports for HuggingFace, LLaMA-Factory, Axolotl, OpenAI, and custom RAG stacks. The workspace bundles the Rust core, CLI, doc2dataset pipeline, HTTP service + UI, and Python/Node bindings. Documentation - Research paper (PDF) - Technical Report / Spec - CLI guide - Configuration guide - Data format reference - Installation notes - Evaluation data (GitHub Releases) – evaluation corpora and metrics are distributed as a GitHub Release asset (see the latest release for a 3dcf-eval- .tar. archive). Download the archive, unpack it at the repo root (it recreates the eval/ tree), and then follow eval/README.md . Features - Document layer standard – deterministic macro-cells with kind / bbox / importance stored in three JSONL files: documents.jsonl , pages.jsonl , and cells.jsonl . - NumGuard numeric integrity – per-cell number hashes with A/B/C/D coverage; in our evaluation, all A-bucket corruptions are detected (recall 1.0). - Token-efficient contexts – macro-cell contexts are typically 3–6× smaller in tokens than naive pdfminer/Unstructured baselines on our micro-corpora, while maintaining or improving QA accuracy","default_branch":null,"files":null,"tree":[],"storefront":"/r/3DCF-Labs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/3DCF-Labs/doc2dataset/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}