{"repo":"viggy28/streambed","free":true,"listed":false,"github":"https://github.com/viggy28/streambed","clone":"git clone https://github.com/viggy28/streambed.git","description":"Stream Postgres to Apache Iceberg on S3 via logical replication, queryable over the Postgres wire protocol.","language":"Go","stars":266,"topics":["duckdb","iceberg","parquet","postgres","s3"],"license":"Apache-2.0","category":"databases-storage","readme_excerpt":"Streambed Postgres-to-Iceberg CDC engine. Offload analytical queries from your production database without changing your application. streambed streams WAL changes via logical replication, writes Parquet files to S3, and commits Iceberg metadata. Query the result with any Iceberg-compatible engine -- or use the built-in query server, which speaks the Postgres wire protocol so you can connect with psql . See It In Action Same analytical query on pgbench (1M accounts, 500K history rows). Postgres on the left, Streambed on the right. No ETL. No Spark. Just Postgres + S3. Quick Start Run streambed sync --help for all configuration options. All flags support environment variables with STREAMBED prefix (e.g. STREAMBED SOURCE URL ). UPDATE/DELETE use copy-on-write by default; opt into Iceberg v2 equality deletes with --mutation-mode=mor (or STREAMBED MUTATION MODE=mor ) after verifying reader compatibility. Once a table has active equality deletes, Streambed fails startup in COW mode; continue using MOR. Use --target-file-size-mb (or STREAMBED TARGET FILE SIZE MB ) to split large flushes into approximately target-sized Parquet data files. The default is 128 MiB. This is a target, not a hard maximum; small flushes still create small files. Architecture How It Works Streambed connects to Postgres as a logical replication subscriber. It decodes WAL messages (inserts, updates, deletes), buffers rows per table, and periodically flushes them as Parquet files to S3 with Iceberg metadata co","default_branch":null,"files":null,"tree":[],"storefront":"/r/viggy28","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/viggy28/streambed/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}