{"repo":"omshdev/Rafka","free":true,"listed":false,"github":"https://github.com/omshdev/Rafka","clone":"git clone https://github.com/omshdev/Rafka.git","description":"Rafka is a lightweight, re-implementation of Apache Kafka written in Rust, built from the ground up to understand how distributed commit logs work. It aims to explore the internal mechanics of Kafka — including durability, partitioning, offsets, replication, and fault-tolerant message delivery.","language":"Rust","stars":10,"topics":["kafka","rust","rust-lang"],"license":null,"category":"data-pipelines","readme_excerpt":"🦀 Rafka — A Distributed Commit Log in Rust Rafka is a lightweight, educational re-implementation of Apache Kafka written in Rust , built from the ground up to understand how distributed commit logs work. It aims to explore the internal mechanics of Kafka — including durability, partitioning, offsets, replication, and fault-tolerant message delivery. --- 🚀 Overview Rafka is not a library — it’s a learning project that mimics the essential building blocks of Kafka’s architecture: Persistent append-only commit log stored on disk Topic-based message organization Partitions for scalability and ordering Producers that append records Consumers that read from offsets A simple TCP-based broker handling commands like PRODUCE and CONSUME --- 🎯 Project Goals 🧱 Build a durable append-only log for message persistence ⚙️ Implement topic and partition management 🔁 Add replication and leader election (in future versions) 📨 Support a basic producer–consumer protocol over TCP 🧠 Deeply understand Kafka’s internals — such as segment files , indexes , message formats , commit logs , and offset tracking --- 🧩 Current Features ✅ In-memory topic management with file-based log persistence ✅ Partitioned message storage (1..N partitions per topic) ✅ PRODUCE / CONSUME TCP commands ✅ Offset-based message reading ✅ Thread-safe broker using Arc ✅ Disk-backed logs under data/ /partition- .log ✅ Testable with simple clients like nc or a custom Rust client --- 🧠 Example Usage Start the Broker Output: ","default_branch":null,"files":null,"tree":[],"storefront":"/r/omshdev","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/omshdev/Rafka/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}