{"repo":"nubskr/walrus","free":true,"listed":false,"github":"https://github.com/nubskr/walrus","clone":"git clone https://github.com/nubskr/walrus.git","description":"🦭 Distributed log streaming engine built from first principles","language":"Rust","stars":1920,"topics":["database","rust","kafka","apache-kafka","nats","redpanda","storage-engine","stream-processing","streams"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Walrus: A Distributed Message Streaming Engine Walrus is a distributed message streaming platform built on a high-performance log storage engine. It provides fault-tolerant streaming with automatic leadership rotation, segment-based partitioning, and Raft consensus for metadata coordination. Key Features: - Automatic load balancing via segment-based leadership rotation - Fault tolerance through Raft consensus (3+ nodes) - Simple client protocol (connect to any node, auto-forwarding) - Sealed segments for historical reads from any replica - High-performance storage with io uring on Linux Architecture System Overview Producers and consumers connect to any node (or via load balancer). The cluster automatically routes requests to the appropriate leader and manages segment rollovers for load distribution. Node Internals Each node contains four key components: Node Controller (routing and lease management), Raft Engine (consensus for metadata), Cluster Metadata (replicated state), and Bucket Storage (Walrus engine with write fencing). Core Components Node Controller - Routes client requests to appropriate segment leaders - Manages write leases (synced from cluster metadata every 100ms) - Tracks logical offsets for rollover detection - Forwards operations to remote leaders when needed Raft Engine (Octopii) - Maintains Raft consensus for metadata changes only (not data!) - Handles leader election and log replication - Syncs metadata across all nodes via AppendEntries RPCs Cluster Met","default_branch":null,"files":null,"tree":[],"storefront":"/r/nubskr","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/nubskr/walrus/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}