{"repo":"danthegoodman1/icedb","free":true,"listed":false,"github":"https://github.com/danthegoodman1/icedb","clone":"git clone https://github.com/danthegoodman1/icedb.git","description":"An in-process Parquet merge engine for better data warehousing in S3 with MVCC","language":"Python","stars":153,"topics":["database","duckdb","serverless","python","data","parquet","warehouse"],"license":null,"category":"databases-storage","readme_excerpt":"IceDB IceDB is an in-process Parquet merge engine for better data warehousing in S3, using only S3, with MVCC, time travel queries, and atomic operations. No Spark, no JVM, no data warehouse experience required. IceDB runs stateless, and stores data in easily readable formats to allow any language or framework to parse the log (jsonl) and read the data (parquet), making it far easier to insert, migrate, query, run, and scale than alternatives. It's queryable by anything that understands parquet, and runs 54x cheaper than managed solutions such as BigQuery, Snowflake, and Athena. IceDB tracks table schemas as standard SQL types, supports dynamic schema evolution, and divides your data into tables and partitions. IceDB merges parquet files and manages tombstone cleanup to optimize your data storage for faster queries, very similar to what systems like ClickHouse do under the hood. Except IceDB does it effectively stateless: All state and storage is in S3. This makes it extremely easy to run and scale. When you need to introduce concurrent mutations at the table level, you can introduce coordination through exclusive locks. The IceDB log format also uses the widely understood newline-delimited JSON format, making it trivial to read from any language. It retains many of the features of modern OLAP systems (such as ClickHouse’s Materialized Views), adds some new ones, and makes it way easier to build scalable data systems with a focus on true multi-tenancy. IceDB can replace syste","default_branch":null,"files":null,"tree":[],"storefront":"/r/danthegoodman1","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/danthegoodman1/icedb/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}