{"repo":"Netflix/genie","free":true,"listed":false,"github":"https://github.com/Netflix/genie","clone":"git clone https://github.com/Netflix/genie.git","description":"Distributed Big Data Orchestration Service","language":"Java","stars":1767,"topics":["big-data","bigdata","orchestration","configuration","configuration-management","java","spring-boot","distributed-systems","netflixoss","cloud"],"license":"Apache-2.0","category":"workflow-automation","readme_excerpt":"Genie Introduction Genie is a federated Big Data orchestration and execution engine developed by Netflix. Genie’s value is best described in terms of the problem it solves. Big Data infrastructure is complex and ever-evolving. Data consumers (Data Scientists or other applications) need to jump over a lot of hurdles in order to run a simple query: - Find, download, install and configure a number of binaries, libraries and tools - Point to the correct cluster, using valid configuration and reasonable parameters, some of which are very obscure - Manually monitor the query, retrieve its output What works today, may not work tomorrow. The cluster may have moved, the binaries may no longer be compatible, etc. Multiply this overhead times the number of data consumers, and it adds up to a lot of wasted time (and grief!). Data infrastructure providers face a different set of problems: - Users require a lot of help configuring their working setup, which is not easy to debug remotely - Infrastructure upgrades and expansion require careful coordination with all users Genie is designed to sit at the boundary of these two worlds, and simplify the lives of people on either side. A data scientist can “rub the magic lamp” and just say “Genie, run query ‘Q’ using engine SparkSQL against production data”. Genie takes care of all the nitty-gritty details. It dynamically assembles the necessary binaries and configurations, execute the job, monitors it, notifies the user of its completion, and mak","default_branch":null,"files":null,"tree":[],"storefront":"/r/Netflix","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Netflix/genie/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}