{"repo":"Netflix/maestro","free":true,"listed":false,"github":"https://github.com/Netflix/maestro","clone":"git clone https://github.com/Netflix/maestro.git","description":"Maestro: Netflix’s Workflow Orchestrator","language":"Java","stars":3823,"topics":["analytics","automation","batch-processing","dag","data-engineering","data-ops","data-orchestrator","data-pipelines","data-science","elt"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Maestro =================================== Maestro is a general-purpose workflow orchestrator that provides a fully managed workflow-as-a-service (WAAS) to the data platform users at Netflix. It serves thousands of users, including data scientists, data engineers, machine learning engineers, software engineers, content producers, and business analysts, for various use cases. It schedules hundreds of thousands of workflows, millions of jobs every day and operates with a strict SLO even when there are spikes in the traffic. Maestro is highly scalable and extensible to support existing and new use cases and offers enhanced usability to end users. You can read more details about it in our series of blog posts - Maestro: Data/ML Workflow Orchestrator at Netflix - Orchestrating Data/ML Workflows at Scale With Netflix Maestro - 100X Faster: How We Supercharged Netflix Maestro's Workflow Engine - Incremental Processing using Netflix Maestro and Apache Iceberg Get started Prerequisite - Git - Java 21 - Gradle - Docker Build it - ./gradlew build Run it - ./gradlew bootRun Run it with AWS module - docker compose -f maestro-aws/docker-compose.yml up - ./gradlew bootRun --args='--spring.profiles.active=aws' Create a sample workflow - curl --header \"user: tester\" -X POST 'http://127.0.0.1:8080/api/v3/workflows' -H \"Content-Type: application/json\" -d @maestro-server/src/test/resources/samples/sample-dag-test-1.json Get the sample workflow definition - curl -X GET 'http://127.0.0.1:8080/api","default_branch":null,"files":null,"tree":[],"storefront":"/r/Netflix","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Netflix/maestro/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}