{"repo":"dataforgelabs/dataforge-core","free":true,"listed":false,"github":"https://github.com/dataforgelabs/dataforge-core","clone":"git clone https://github.com/dataforgelabs/dataforge-core.git","description":"DataForge helps data teams write functional transformation pipelines by leveraging software engineering principles","language":"PLpgSQL","stars":61,"topics":["analytics","data-modeling","elt","elt-pipeline","etl","etl-framework"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"DataForge helps data analysts and engineers build and extend data solutions by leveraging modern software engineering principles. Understanding DataForge DataForge enables writing of inline functions using single-column SQL expressions rather than CTEs, procedural scripts, or set-based models. For an overview of the underlying concepts, check out this introduction blog. Each function: - is pure, with no side effects) - returns single column - is composable with other functions DataForge software engineering principles: - Functional Programming - Inversion of Control - Single Responsibility Principal - DRY These principles allow DataForge projects to be easy to modify and extend - even with thousands of integrated pipelines. Explore the Core CLI or learn more about how Core powers DataForge Cloud. Requirements Dataforge Core is a code framework and command line tool to develop transformation functions and compile them into executable Spark SQL. To run the CLI you will need: - Java 8 or higher - Amazon Corretto is a great option - A PostgreSQL v14+ server with a dedicated empty database - Check out our friends over at Tembo - Python version 3.12+ - Official Link The CLI also includes an integration to run the code in Databricks. To support this you need: - Databricks Workspace - Databricks SQL Warehouse - Developer Personal Access Token Installation and Quickstart - Open a new command line window - Validate Java and Python are installed correctly: - Install Dataforge by running","default_branch":null,"files":null,"tree":[],"storefront":"/r/dataforgelabs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/dataforgelabs/dataforge-core/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}