{"repo":"Trendyol/go-dcp-kafka","free":true,"listed":false,"github":"https://github.com/Trendyol/go-dcp-kafka","clone":"git clone https://github.com/Trendyol/go-dcp-kafka.git","description":"The Go implementation of the Couchbase to Kafka with DCP.","language":"Go","stars":69,"topics":["couchbase","kafka","go","golang","kafka-connect","dcp"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Go Dcp Kafka Go implementation of the Kafka Connect Couchbase. Go Dcp Kafka streams documents from Couchbase Database Change Protocol (DCP) and publishes Kafka events in near real-time. Features Less resource usage and higher throughput (see Benchmarks). Custom Kafka key and headers implementation(see Example). Sending multiple Kafka events for a DCP event (see Example). Handling different DCP events such as expiration, deletion and mutation (see Example). Kafka compression support(Gzip, Snappy, Lz4, Zstd). Kafka producer acknowledges support(fire-and-forget, wait for the leader, wait for the full ISR). Metadata can be saved to Couchbase, Kafka or Connect . Managing batch configurations such as maximum batch size, batch bytes, batch ticker durations. Scale up and down by custom membership algorithms(Couchbase, KubernetesHa, Kubernetes StatefulSet or Static, see examples). Easily manageable configurations . Benchmarks The benchmark was made with the 1,001,006 Couchbase document, because it is possible to more clearly observe the difference in the batch structure between the two packages. Default configurations for Java Kafka Connect Couchbase used for both connectors. Package Time to Process Events Average CPU Usage(Core) Average Memory Usage :------------------------------------ :----------------------: :-----------------------: :--------------------: Go Dcp Kafka (1.20) 12s 0.383 428MB Java Kafka Connect Couchbase(JDK11) 19s 1.5 932MB Example Struct Config File Config File C","default_branch":null,"files":null,"tree":[],"storefront":"/r/Trendyol","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Trendyol/go-dcp-kafka/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}