{"repo":"quangh33/Go-Kafka","free":true,"listed":false,"github":"https://github.com/quangh33/Go-Kafka","clone":"git clone https://github.com/quangh33/Go-Kafka.git","description":"A Distributed, Fault-Tolerant Message Queue from Scratch. Inspired by Apache Kafka","language":"Go","stars":89,"topics":["kafka","message-broker","message-queue"],"license":null,"category":"data-pipelines","readme_excerpt":"Go-Kafka: A Distributed, Fault-Tolerant Message Queue from Scratch Go-Kafka is a from-scratch implementation of a distributed, replicated message queue inspired by Apache Kafka. This project is a deep dive into the core principles of distributed systems, demonstrating concepts like distributed consensus, replication, fault tolerance, and high-performance storage techniques, all built in pure Go. It is not intended to be a production-ready replacement for Kafka, but rather a powerful demonstration of the engineering principles required to build such a system. Core Features - Replicated, Append-Only Commit Log : At its heart is a durable, high-performance storage engine that writes messages to an immutable, append-only log file on disk. This log is the source of truth for all data. - Topics & Partitions : The system supports organizing data into topics, which are subdivided into partitions. This allows for data isolation, multi-tenancy, and serves as the fundamental unit of parallelism for consumers. - Distributed Consensus with Raft : The cluster uses the industry-standard Raft consensus algorithm (via hashicorp/raft) to ensure all brokers agree on the state of the data. This provides: - Automatic Leader Election : For each partition, one broker is elected leader to handle all writes. - Log Replication : The leader replicates all messages to follower nodes. - Fault Tolerance : The cluster can survive the failure of a minority of nodes without downtime or data loss. A new leade","default_branch":null,"files":null,"tree":[],"storefront":"/r/quangh33","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/quangh33/Go-Kafka/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}