{"repo":"zerodha/kaf-relay","free":true,"listed":false,"github":"https://github.com/zerodha/kaf-relay","clone":"git clone https://github.com/zerodha/kaf-relay.git","description":"Replicate and sync Kafka topics between clusters in realtime. Supports topic re-mapping, healthchecks, and hot failovers for high availability.","language":"Go","stars":99,"topics":["kafka","kafka-consumer","kafka-producer","kafka-replicator","franz-go","high-availability"],"license":"MIT","category":"data-pipelines","readme_excerpt":"kaf-relay kaf-relay is an opinionated, high performance program for keeping Kafka clusters in sync by replicating topics. It is specfically designed for high-availability with background healthchecks, offset tracking, and topic lag checks. Features Topic Forwarding: Relay consumes messages from topics in one Kafka cluster and forwards them to topics in another Kafka cluster. Authentication: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 Topic Remapping: Relay allows you to specify topic remappings, enabling you to map a topic from the source cluster to a different topic in the destination cluster. Consumer group failover: Given identical Kafka instances (separate nodes 1...N) at the upstream, instantly switch over to the next node in a round-robin fashion on current node failure. Offset tracking between source and target nodes allows de-duplication without external stores. Topic lag failover: Monitors offsets amongst N identical nodes to detect lags and to instantly switch upstream consumer nodes. Stop at end: Flag --stop-at-end allows the program to stop after reaching the end of consumer topic offsets that was picked up on boot. Filter messages using go plugins: Flag --filter allows the program to filter messages based on the logic in plugin code. kaf-relay in different modes Usage To run kaf-relay, follow these steps: Copy config.sample.toml to config.toml and edit it accordingly. Then run: Filter plugins Plugins allow to parse and filter incoming messages to allow or deny them from ","default_branch":null,"files":null,"tree":[],"storefront":"/r/zerodha","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/zerodha/kaf-relay/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}