{"repo":"deepseek-ai/smallpond","free":true,"listed":false,"github":"https://github.com/deepseek-ai/smallpond","clone":"git clone https://github.com/deepseek-ai/smallpond.git","description":"A lightweight data processing framework built on DuckDB and 3FS.","language":"Python","stars":4991,"topics":["data-processing","duckdb"],"license":"MIT","category":"data-pipelines","readme_excerpt":"smallpond A lightweight data processing framework built on [DuckDB] and [3FS]. Features - 🚀 High-performance data processing powered by DuckDB - 🌍 Scalable to handle PB-scale datasets - 🛠️ Easy operations with no long-running services Installation Python 3.8 to 3.12 is supported. Quick Start Documentation For detailed guides and API reference: - Getting Started - API Reference Performance We evaluated smallpond using the [GraySort benchmark] ([script]) on a cluster comprising 50 compute nodes and 25 storage nodes running [3FS]. The benchmark sorted 110.5TiB of data in 30 minutes and 14 seconds, achieving an average throughput of 3.66TiB/min. Details can be found in [3FS - Gray Sort]. [DuckDB]: https://duckdb.org/ [3FS]: https://github.com/deepseek-ai/3FS [GraySort benchmark]: https://sortbenchmark.org/ [script]: benchmarks/gray sort benchmark.py [3FS - Gray Sort]: https://github.com/deepseek-ai/3FS?tab=readme-ov-file#2-graysort Development License This project is licensed under the MIT License.","default_branch":null,"files":null,"tree":[],"storefront":"/r/deepseek-ai","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/deepseek-ai/smallpond/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}