{"repo":"sspaeti/clickhouse-modeling-rill-example","free":true,"listed":false,"github":"https://github.com/sspaeti/clickhouse-modeling-rill-example","clone":"git clone https://github.com/sspaeti/clickhouse-modeling-rill-example.git","description":"Data Modeling Guide: ClickHouse ETL with NOAA Weather Data Example","language":null,"stars":12,"topics":["clickhouse","data-engineering","data-visualization","data-engineering-p"],"license":null,"category":"data-pipelines","readme_excerpt":"Data Modeling Guide: ClickHouse ETL with NOAA Weather Data Example We use S3 source with ClickHouse as engine, and Rill for visualization. This example demonstrates advanced data modeling strategies for real-time analytics using ClickHouse and Rill. We showcase how to maximize ClickHouse's capabilities for data transformation while keeping Rill focused on visualization and last-mile analytics. Note This is a clone of Rill Examples and added for Weather-demo data please check the readme for how to install and setup with env variables Quick Installation Contains This project contains example Rill project with: - ClickHouse configured as the default OLAP engine - An incremental source that ingests Hive-partitioned Parquet files from S3 - A source that ingests a lookup table from Postgres and creates it as a ClickHouse dictionary - A metrics view with a dimension that does a query-time lookup in the dictionary table Architecture Overview Data Flow and Incremental Loading Behavior How Rill Manages ClickHouse Data When you run rill start , here's what happens: 1. ClickHouse Instance : Rill starts a managed ClickHouse server (local development mode) 2. Partition Discovery : Rill evaluates the partition SQL: SELECT arrayJoin(['2020', '2021', '2022']) AS year 3. Incremental Loading : For each partition, Rill checks if data exists and is current 4. Data Persistence : ClickHouse stores transformed data in MergeTree tables that persist across restarts Incremental Strategy: partition over","default_branch":null,"files":null,"tree":[],"storefront":"/r/sspaeti","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/sspaeti/clickhouse-modeling-rill-example/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}