{"repo":"beebeeep/chafka","free":true,"listed":false,"github":"https://github.com/beebeeep/chafka","clone":"git clone https://github.com/beebeeep/chafka.git","description":"Real-time Kafka to ClickHouse ingestion service","language":"Rust","stars":35,"topics":["clickhouse","ingestion","kafka"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Chafka ====== Extensible service for real-time data ingestion from Kafka to ClickHouse. Motivation ========== ClickHouse can natively consume data from Kafka using Kafka table engine, or Kafka Connect sink. However, these solutions have own issues - in particular, lack of flexibility and observability. This project is a standalone service that allows you to ingest your data in safe and controllable manner while providing full control over schema and deserialization process. Installation ============ Use cargo. Configuration ============= Example of the config can be found in example.toml. Architecture ============ Conceptually, service consists of two main components - the ingestion core, responsible for consuming messages from Kafka and writing data to ClickHouse; and one or several \"decoders\" - packages implementing a simple trait to unmarshal message from Kafka into set of ClickHouse columns. Out of the box there is a universal configurable Avro decoder, and you also may add your own. Kafka and ClickHouse ==================== Chafka uses Kafka's consumer groups and performs safe offset management - it will only commit offsets of messages that have been successfully inserted into CH. Chafka also automatically batches inserts to CH for optimal performance. Batching is controlled by batch size and batch timeout, allowing user to tune ingestion process either for throughput or for latency. Delivery and consistency guarantees =================================== The baseline is ","default_branch":null,"files":null,"tree":[],"storefront":"/r/beebeeep","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/beebeeep/chafka/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}