{"repo":"starlake-ai/starlake","free":true,"listed":false,"github":"https://github.com/starlake-ai/starlake","clone":"git clone https://github.com/starlake-ai/starlake.git","description":"Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.","language":"Scala","stars":210,"topics":["spark","bigquery","hdfs","redshift","snowflake","synapse","etl","data-engineering","data-integration","data-pipeline"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Declarative Data Pipelines. Extract. Load. Transform. Orchestrate. Documentation • Installation • Data Stacks • Contributing --- Starlake replaces hundreds of lines of BigQuery/Snowflake/Redshift/Spark/SQL boilerplate with simple YAML declarations. Define what your data pipeline should do — Starlake figures out how . Inspired by Terraform and Ansible, Starlake brings declarative programming to data engineering: schema inference, merge strategies, data quality checks, lineage tracking, and DAG generation — all from configuration files. Why Starlake? - No code, just config - YAML declarations replace custom ETL scripts - Any warehouse - BigQuery, Snowflake, Redshift, DuckDB, PostgreSQL, Delta Lake, Iceberg - Any orchestrator - Airflow, Dagster, Snowflake Tasks with auto-generated DAGs - Any source - JDBC databases, CSV, JSON, XML, fixed-width, Parquet, Kafka - Schema inference - Auto-detect formats, headers, separators, and data types - Built-in data quality - Expectations and validation at load time - Data lineage - Automatic dependency tracking across your entire pipeline - Privacy controls - Column-level encryption and access policies Quick Start macOS / Linux: Windows (PowerShell): Docker: Then: For pre-built production-ready data stacks, see Starlake Pragmatic Data Stacks. How It Works 1. Extract Pull data from any JDBC source with a few lines of YAML: 2. Load Define schemas, merge strategies, and data quality rules: 3. Transform Write SQL, Starlake generate","default_branch":null,"files":null,"tree":[],"storefront":"/r/starlake-ai","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/starlake-ai/starlake/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}