{"repo":"jairus-m/dagster-sdlc","free":true,"listed":false,"github":"https://github.com/jairus-m/dagster-sdlc","clone":"git clone https://github.com/jairus-m/dagster-sdlc.git","description":"An e2e pipeline using dlt, dagster, duckdb, and dbt-core","language":"Python","stars":24,"topics":["dagster","dbt","dlthub","duckdb","sklearn","snowflake","software-development-lifecycle","uv"],"license":null,"category":"data-pipelines","readme_excerpt":"The Analytics Development Lifecycle within a Modern Data Engineering Framework Utilizing dltHub, dbt, + Dagster as a framework for developing data products with software engineering best practices. While the short-term goal is to learn these tools, the greater goal is to understand and flesh out what the full development and deployment cycle can look like for orchestrating a data platform and deploying custom pipelines. There is a great process in the transformation layer using dbt where we have local development, testing, versioning/branching, CICD, code-review, separation of dev and prod, project structure/cohesion etc., but how can we apply that to the entire data platform and espeacially, the 10-20% of ingestion jobs that cannot be done in a managed tool like Airbyte and/or is best done using a custom solution? Current Status Dagster - Orchestrated ingest, transformation, and downstream dependecies (ML/Analytics) with Dagster - https://github.com/jairus-m/dagster-dlt/pull/2, https://github.com/jairus-m/dagster-dlt/pull/6 - Developed in dev environment and materaizlied in dagster dev server - Configured resources / credentials in a root .env file - Current Dagster folder structure (dependencies managed by UV) - https://github.com/jairus-m/dagster-dlt/pull/15 - One code location: dagster proj/ - Assets: dagster proj/assets/ - Resources: dagster proj/resources/ init .py - Jobs: dagster proj/jobs/ init .py - Schedules: dagster proj/schedules/ init .py - Utils: dagster proj/ut","default_branch":null,"files":null,"tree":[],"storefront":"/r/jairus-m","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jairus-m/dagster-sdlc/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}