{"repo":"sky-uk/kafka-message-scheduler","free":true,"listed":false,"github":"https://github.com/sky-uk/kafka-message-scheduler","clone":"git clone https://github.com/sky-uk/kafka-message-scheduler.git","description":"Scheduler for low-frequency and long-term scheduling of delayed messages to Kafka topics.","language":"Scala","stars":35,"topics":["kafka","scheduler"],"license":"BSD-3-Clause","category":"data-pipelines","readme_excerpt":"Kafka Message Scheduler This application is a scheduler for low-frequency and long-term scheduling of delayed messages to Kafka topics. Background This component was initially designed for Sky's metadata ingestion pipeline. We wanted to manage content expiry (for scheduled airings or on-demand assets) in one single component, instead of implementing the expiry logic on all consumers. Given that the pipeline is based on Kafka, it felt natural to use it as input, output and data store. How it works The Kafka Message Scheduler (KMS for short) consumes messages from configured source (schedule) topics. On this topic: - message keys are \"Schedule IDs\" - string values, with an expectation of uniqueness - message values are Schedule messages, encoded in Avro binary format according to the Schema. A schedule is composed of: - The topic you want to send the delayed message to - The timestamp telling when you want that message to be delivered - The actual message to be sent, both key and value The KMS is responsible for sending the actual message to the specified topic at the specified time. Note If the timestamp of when to deliver the message is in the past, the schedule will be sent immediately. The Schedule ID can be used to delete a scheduled message, via a delete message (with a null message value) in the source topic. Startup logic When the KMS starts up it uses the kafka-topic-loader to consume all messages from the configured schedule-topics and populate the scheduling actors s","default_branch":null,"files":null,"tree":[],"storefront":"/r/sky-uk","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/sky-uk/kafka-message-scheduler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}