{"repo":"pujansrt/data-genie","free":true,"listed":false,"github":"https://github.com/pujansrt/data-genie","clone":"git clone https://github.com/pujansrt/data-genie.git","description":"High performant ETL engine written in TypeScript","language":"TypeScript","stars":16,"topics":["data-processing","etl","etl-framework","streaming-data","typescript"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Data-Genie 🧞‍♂️ A high-performant, streaming-first ETL Engine for Node.js and TypeScript , designed for processing massive datasets with a constant memory footprint . --- Documentation & Examples Visit our full documentation site for in-depth guides, API reference, and real-world recipes: https://pujansrt.github.io/data-genie/ --- Installation Note: zod , @aws-sdk/client-s3 , and exceljs are optional peer dependencies in case you need them. Quick Start Preview (Dry Run) Verify your transformations and filters instantly without writing any data. --- Interactive Tools Build your pipelines visually without writing code from scratch: - Declarative Pipeline Builder : Visually configure your YAML pipelines and copy the generated config. - TypeScript Code Generator : Generate full TypeScript boilerplate for complex ETL tasks (S3, SQL, Validation, etc.). --- Why Data-Genie? (Performance Benchmark) In our latest benchmarks (Processing 500k records), Data-Genie used 100x less memory than standard array-based processing. Data Size Naive Approach (Array-based) Data-Genie (Streaming) :--- :--- :--- 100 KB 10 MB RAM 10 MB RAM 100 MB 150 MB RAM 12 MB RAM 10 GB CRASH (OOM) 15 MB RAM --- Features - Streaming-First: Constant memory footprint regardless of file size (O(1) memory complexity). - Multi-Format: Support for CSV, TSV, JSON, NDJSON, Parquet, Excel, and SQL. - Transport Agnostic: Read/Write from Local Disk, AWS S3, HTTP APIs, or Memory. - Fault Tolerant: Retries, Circuit Breakers, and","default_branch":null,"files":null,"tree":[],"storefront":"/r/pujansrt","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/pujansrt/data-genie/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}