{"repo":"pzhenzhou/lakeflow-controller","free":true,"listed":false,"github":"https://github.com/pzhenzhou/lakeflow-controller","clone":"git clone https://github.com/pzhenzhou/lakeflow-controller.git","description":"k8s operator for declarative data pipeline","language":"Go","stars":10,"topics":["data-pipeline","workflow","data-pipline-orchestration"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"lakeflow-controller A Kubernetes operator for data-processing workflows. It exposes a single declarative API — the LakeFlow custom resource ( lakeflow.io/v1alpha1 ) — and compiles it into the underlying engine resources, so you describe what you want to run and the controller manages how it runs on the cluster. Why lakeflow-controller Running data pipelines on Kubernetes usually means stitching together several systems by hand: Argo Workflows for the DAG, Spark Operator for Spark jobs, Volcano for batch scheduling, plus all the storage, retry, and lifecycle glue in between. Each has its own API, and keeping them consistent is tedious and error-prone. lakeflow-controller collapses that into one resource you actually care about — the workflow . You write a LakeFlow describing your tasks and how they depend on each other; the controller generates and reconciles the right Argo/Spark/Volcano objects, wires up storage and credentials, and keeps their status in sync. Concretely, it gives you: - One declarative surface instead of three or four engine-specific APIs. - A data-oriented task model ( spark / python / bash executors) rather than raw pod templates — common concerns like Spark config layering, queues, and local-dir storage are built in. - Built-in operational behavior — retries with backoff, resubmit/retry of failed runs, and suspend/stop/resume/terminate lifecycle control. - No lock-in to a custom engine — it orchestrates standard, battle-tested projects, so your workloads ","default_branch":null,"files":null,"tree":[],"storefront":"/r/pzhenzhou","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/pzhenzhou/lakeflow-controller/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}