{"repo":"fyndiq/confluent_kafka_helpers","free":true,"listed":false,"github":"https://github.com/fyndiq/confluent_kafka_helpers","clone":"git clone https://github.com/fyndiq/confluent_kafka_helpers.git","description":null,"language":"Python","stars":14,"topics":["confluent","kafka","python","client","helpers"],"license":null,"category":"data-pipelines","readme_excerpt":"Confluent Kafka helpers Library built on top of Confluent Kafka Python adding abstractions for consuming and producing messages in a more Pythonic way. Graceful shutdown Importing confluent kafka helpers installs handlers for SIGTERM and SIGINT that do not exit the process immediately . Instead the in-flight message handler always runs to completion before the loop exits. Chaining with other signal handlers If another library (ddtrace, OpenTelemetry SDK, gunicorn, uvicorn, ...) has already installed a SIGTERM/SIGINT handler before confluent kafka helpers is imported, the existing handler is captured and still called after the flag is set. Polling the flag from your own code Long-running handlers can check the flag and bail out early between subtasks: Operational notes - SIGKILL cannot be intercepted. Kubernetes will still send it after terminationGracePeriodSeconds if the process hasn't exited. The default value for this in k8s is 30 seconds. If handling a single message can take longer than that the application need to configure a longer terminationGracePeriodSeconds in the k8s manifest - The flag is process-wide. Tests that exercise it must clear it (see tests/test signals.py for the autouse reset fixture pattern). OpenTelemetry (OTEL) Test generation of spans Make sure you have opentelemetry-sdk installed. Add this code to your applications entry point as early as possible, e.g: All spans will now be printed to the std out instead of getting exported. NOTE! This will most ","default_branch":null,"files":null,"tree":[],"storefront":"/r/fyndiq","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/fyndiq/confluent_kafka_helpers/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}