{"repo":"ben-jamin-chen/kafka-streams-redis-statestore","free":true,"listed":false,"github":"https://github.com/ben-jamin-chen/kafka-streams-redis-statestore","clone":"git clone https://github.com/ben-jamin-chen/kafka-streams-redis-statestore.git","description":"A demo project demonstrating how to read and write data into a Redis-backed state store using Kafka Streams (4.3.1)","language":"Java","stars":24,"topics":["spring-boot","java","restful-api","kafka-streams","springdoc-openapi","swagger-ui","docker-compose","openapi3","avro","redis"],"license":"MIT","category":"databases-storage","readme_excerpt":"Kafka Streams (4.3.1) with a Redis-backed State Store By default, Kafka Streams utilizes the RocksDB storage engine for persistent state stores. One glaring issue you may have noticed is the biggest delay occurs when Kafka Streams is in a rebalancing state, where it internally rebuilds these state stores from the change-log topics. This is especially a common problem if you use Kubernetes and your environment is designed to scale up (new pods) or down (nodes are removed). While it’s in a rebalancing state, all involved consumer processing is blocked, so what this means is the application won’t be able to service any requests until the process becomes unblocked. This isn’t ideal especially in a production environment or from a practicality standpoint. The overall time with this delay seems to correspond with how large your dataset is in your topics, so if you're developing under a continuous integration environment this poses a major challenge since every deployment will require some amount of upfront time to rebalance. Anyways, this is the primary motivation for switching over to a custom state store. We can leverage a permanent storage solution like Redis such that in any new deployment or application restart, we can avoid these upfront rebalancing delay. What You Need Java 25 Maven 3.9+ (or use the included Maven wrapper ./mvnw ) Docker 20+ (with Docker Compose v2) Redis 7+ Getting Started First, install a local instance of Redis (if you haven't already). If you're running ","default_branch":null,"files":null,"tree":[],"storefront":"/r/ben-jamin-chen","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ben-jamin-chen/kafka-streams-redis-statestore/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}