{"repo":"osodevops/k2i","free":true,"listed":false,"github":"https://github.com/osodevops/k2i","clone":"git clone https://github.com/osodevops/k2i.git","description":"K2I - Kafka to Iceberg streaming ingestion engine. A Rust CLI tool inspired by Moonlink architecture that consumes from Kafka, buffers with Apache Arrow for sub-second query freshness, and writes to Apache Iceberg tables.","language":"Rust","stars":23,"topics":["apache-arrow","apache-iceberg","cli","data-engineering","data-ingestion","iceberg","kafka","real-time","rust","streaming"],"license":null,"category":"data-pipelines","readme_excerpt":"K2I: Kafka to Apache Iceberg in One Rust Binary Stream final-form Kafka events into Apache Iceberg tables with Protobuf Schema Registry decoding, Arrow hot reads, Parquet writes, and Docker-verified DuckDB/Iceberg validation. --- K2I is an open-source, standalone Kafka-to-Iceberg ingestion engine written in Rust. It consumes a Kafka topic, decodes raw or Confluent-framed Protobuf messages, keeps recent rows visible through an Arrow-backed local read path, and flushes Parquet data files through Iceberg catalog commits. K2I is built for teams that want fresh lakehouse tables without operating a Flink job, Spark micro-batch pipeline, or Kafka Connect cluster for a simple final-form event stream. One process owns one configured Kafka topic and one configured Iceberg table today. Use K2I When - You have final-form Kafka events that should become analytics rows in Apache Iceberg. - You want a single Rust service/container instead of a stream-processing cluster for this ingestion job. - You need local fresh-read visibility before the next Iceberg snapshot is committed. - You use Confluent Schema Registry Protobuf and want additive schema evolution guarded by readiness checks. - You want local Docker E2E validation that proves the written Iceberg table is readable by DuckDB. K2I is not a general stream processing framework. Use Flink or another stream processor for joins, windows, stateful transformations, multi-source ETL, or complex CDC delete/update semantics. Quick Local Proof Ru","default_branch":null,"files":null,"tree":[],"storefront":"/r/osodevops","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/osodevops/k2i/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}