{"repo":"jaehyeon-kim/kafka-pocs","free":true,"listed":false,"github":"https://github.com/jaehyeon-kim/kafka-pocs","clone":"git clone https://github.com/jaehyeon-kim/kafka-pocs.git","description":"Apache Kafka and Related Projects","language":"Shell","stars":32,"topics":["aws","kafka","kafka-connect","msk","msk-connect","python","data-streaming","docker","docker-compose","real-time-analytics"],"license":null,"category":"deployment-docker-iac","readme_excerpt":"Kafka POCs Kafka POC projects - How to configure Kafka consumers to seek offsets by timestamp - Normally, we consume Kafka messages from the beginning/end of a topic, or the last committed offsets. For backfilling or troubleshooting however, we occasionally need to consume messages from a certain timestamp. The Kafka consumer class of the kafka-python package has a method to seek a particular offset for a topic partition. Therefore, if we know which topic partition to choose – such as by assigning a topic partition – we can easily override the fetch offset. When we deploy multiple consumer instances together however, we make them subscribe to a topic, and topic partitions are dynamically assigned, which means we do not know which topic partition will be assigned to a consumer instance in advance. In this post, we will discuss how to configure the Kafka consumer to seek offsets by timestamp where topic partitions are dynamically assigned by subscription. - Simplify Streaming Ingestion on AWS – Part 1 MSK and Redshift - Apache Kafka is a popular distributed event store and stream processing platform. Previously loading data from Kafka into Redshift and Athena usually required Kafka connectors (e.g. Amazon Redshift Sink Connector and Amazon S3 Sink Connector). Recently these AWS services provide features to ingest data from Kafka directly, which facilitates a simpler architecture that achieves low-latency and high-speed ingestion of streaming data. In part 1 of the simplify stre","default_branch":null,"files":null,"tree":[],"storefront":"/r/jaehyeon-kim","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jaehyeon-kim/kafka-pocs/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}