{"repo":"harrydevforlife/building-lakehouse","free":true,"listed":false,"github":"https://github.com/harrydevforlife/building-lakehouse","clone":"git clone https://github.com/harrydevforlife/building-lakehouse.git","description":"Building Data Lakehouse by open source technology. Support end to end data pipeline, from source data on AWS S3 to Lakehouse, visualize and recommend app.","language":"Python","stars":43,"topics":["airflow","dbt","delta-lake","flask-api","hive-metastore","lakehouse","metabase","minio","python","s3"],"license":null,"category":"data-pipelines","readme_excerpt":"Data Lakehouse Open Data Lakehouse Platform with specialized tools for data analytics Table of contents - Table of contents - Description - System Architecture - Lakehouse Architecture - Quick start - Requirements - Installation - Usage - Access services - Prepare data - Orchestrate pipeline - Query data - Clean up - Todo Description Warning : This project is just a poc for open-sourced lakehouse platform. This project is a data lakehouse platform with specialized tools for data analytics. The platform is built on top of the following components: - Apache Spark: Apache Spark is a unified analytics engine for big data processing, with built-in modules for streaming, SQL, machine learning and graph processing. - Delta Lake: Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark and big data workloads. - Apache Airflow: Apache Airflow is a platform to programmatically author, schedule and monitor workflows. - MinIO: MinIO is a high performance, distributed object storage system. It is software-defined, runs on industry standard hardware and is 100% open source under the Apache V2 license. - Metabase: Metabase is the easy, open source way for everyone in your company to ask questions and learn from data. - Hive Metastore: The Hive metastore is a central repository of Hive metadata. It stores metadata for Hive tables (like their schema and location) and partitions in a relational database. - Mlflow: MLflow is an open source platform for the comple","default_branch":null,"files":null,"tree":[],"storefront":"/r/harrydevforlife","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/harrydevforlife/building-lakehouse/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}