{"repo":"apache/hamilton","free":true,"listed":false,"github":"https://github.com/apache/hamilton","clone":"git clone https://github.com/apache/hamilton.git","description":"Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.","language":"Jupyter Notebook","stars":2567,"topics":["data-science","python","dag","data-engineering","dataframe","etl","etl-framework","etl-pipeline","feature-engineering","machine-learning"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Apache Hamilton — portable & expressive data transformation DAGs Disclaimer Apache Hamilton is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. Apache Hamilton (incubating) is a lightweight Python library for directed acyclic graphs (DAGs) of data transformations. Your DAG is portable ; it runs anywhere Python runs, whether it's a script, notebook, Airflow pipeline, FastAPI server, etc. Your DAG is expressive ; Apache Hamilton has extensive features to define and modify the execution of a DAG (e.g., data validation, experiment tracking, remote execution). To create a DAG, write regular Python functions that specify their dependencies with their parameters. As shown below, it results in readable code that can always be visualized. Apache Hamilton loads that definition and automatically builds the DAG for you! Functions B() and C() refer to function A via their parameters Apache Hamilton brings modularity and structure to any Python application moving data: ETL pipelines, ML workflows, LLM applications, RAG systems,","default_branch":null,"files":null,"tree":[],"storefront":"/r/apache","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/apache/hamilton/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}