{"repo":"Trendyol/kafka-retry-job","free":true,"listed":false,"github":"https://github.com/Trendyol/kafka-retry-job","clone":"git clone https://github.com/Trendyol/kafka-retry-job.git","description":"Kafka Retry Job is an open-source project for moving Kafka messages from error topics to retry topics, developed by Trendyol with love :orange_heart:","language":"C#","stars":28,"topics":["kafka"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Introduction Kafka Retry Job is an open-source project for moving Kafka messages from error topics to retry topics, developed by Trendyol with love Why? In event-driven architecture, anything can happen. Sometimes our consumers fail to do their work due to network issues, business validations etc. and we need to retry that specific message. So we developed our own tool, which has proven to be very useful, and we decided to share it with the community. How? Kafka Retry Job is a Kubernetes CronJob. When the job starts, it selects all topics from the Kafka cluster and filters out the error topics via Regex. After that, it gets the messages by partition and moves them to the retry topics. Features - Dockerizability: Can work in a Docker container. - Configurability: The user can specify error topics via Regex. Also can specify Error Suffix and Retry Suffix. - Scalability: The project can work in parallel, so you can deploy to various machine for availablity etc. - Schedulability: Can work at the given interval. - Config Management: You can seperate your configs and secrets in order to secure your passwords. For this purpose you can use and Configs Here is the explanation of environment variables. Please note that the config properties are case sensitive. - BootstrapServers: Kafka Cluster servers. You can give multiple servers with comma - Topic Regex: Regex for filtering error topics - Error Suffix: The suffix of error topics - Retry Suffix: The suffix of retry topics - GroupId: ","default_branch":null,"files":null,"tree":[],"storefront":"/r/Trendyol","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Trendyol/kafka-retry-job/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}