{"repo":"realdatadriven/etlx","free":true,"listed":false,"github":"https://github.com/realdatadriven/etlx","clone":"git clone https://github.com/realdatadriven/etlx.git","description":"ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.","language":"Go","stars":52,"topics":["data-engineering","data-quality","data-quality-checks","data-quality-monitoring","data-science","duckdb","etl","object-storage","relational-databases","report"],"license":"MIT","category":"data-pipelines","readme_excerpt":"ETLX ETLX is an open-source, SQL-first data workflow engine and an evolving specification for building self-documenting data pipelines . Pipelines are defined using structured Markdown , which serves simultaneously as: executable configuration human-readable documentation governance and audit artifacts source ETLX pipelines can be executed , versioned , and rendered as documentation — making the workflow itself the source of truth. It combines: Declarative pipelines Executable documentation Multi-engine SQL execution Built-in observability Powered by DuckDB , but not locked to it . --- ✨ What Makes ETLX Different? - ✔ Pipelines are written in Markdown + YAML + SQL - ✔ The pipeline is the documentation - ✔ Runs on DuckDB, PostgreSQL, SQLite, MySQL, SQL Server, ODBC - ✔ One specification for ETL / ELT / Reverse ETL, data quality, report generation and automation, scripts execution, ... - ✔ Fully auditable & reproducible by design - ✔ Available as a CLI and embeddable Go library ETLX is not just a runtime — it also meant to be a specification for declarative data workflows , where all logic is explicit, inspectable, and versionable . --- 🚀 Quick Example - pipeline.md yaml name: INPUTS description: this defines a ETL / ELT / Reverse ETL block where every level two block with proper metadata (yaml) is treated as a step in the workflow runs as: ETL # the runs as defines how the block shoud be treated active: true # active if missing the is consider active, if false this block and ","default_branch":null,"files":null,"tree":[],"storefront":"/r/realdatadriven","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/realdatadriven/etlx/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}