{"repo":"relytcloud/pg_duckpipe","free":true,"listed":false,"github":"https://github.com/relytcloud/pg_duckpipe","clone":"git clone https://github.com/relytcloud/pg_duckpipe.git","description":"Real-time streaming ingestion (CDC) for HTAP in PostgreSQL","language":"Rust","stars":68,"topics":["analytics","cdc","duckdb","ducklake","htap","postgresql","postgresql-extension","rust"],"license":"MIT","category":"databases-storage","readme_excerpt":"pg duckpipe PostgreSQL extension for real-time CDC to pg ducklake Overview pg duckpipe brings real-time change data capture (CDC) to PostgreSQL, enabling HTAP by continuously syncing your row tables to DuckLake columnar tables. Run transactional and analytical workloads in a single database. Key Features - One-command setup : duckpipe.add table() creates the columnar target and starts syncing automatically - Real-time CDC : streams WAL changes continuously with 5 s default flush latency (tunable) - Sync groups : group multiple tables to share a single publication and replication slot Quick Start This pulls the Docker image, starts PostgreSQL with all extensions loaded, and drops you into psql. Then try: Requirements The Docker image ships everything preconfigured. The notes below apply when installing from source. - PostgreSQL 17 or 18 (tested in CI) - pg ducklake (which bundles pg duckdb and libduckdb) must be installed - wal level = logical with sufficient max replication slots and max wal senders - Both extensions must be preloaded: shared preload libraries = 'pg duckdb, pg duckpipe' - Source tables must have a PRIMARY KEY (required by logical replication to identify rows) Benchmark Sysbench on Apple M1 Pro, 100k rows/table, 30s OLTP phase: Scenario Snapshot OLTP TPS Avg Lag Consistency ---------- ---------- ---------- --------- ------------- 1 table, oltp insert 136,240 rows/s 10,102 3.3 MB PASS 4 tables, oltp insert 163,599 rows/s 9,413 64.6 MB PASS 1 table, oltp read wr","default_branch":null,"files":null,"tree":[],"storefront":"/r/relytcloud","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/relytcloud/pg_duckpipe/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}