{"repo":"Victor-Kipruto-Rop/streaming-pipeline","free":true,"listed":false,"github":"https://github.com/Victor-Kipruto-Rop/streaming-pipeline","clone":"git clone https://github.com/Victor-Kipruto-Rop/streaming-pipeline.git","description":"Real-time data streaming pipeline built with modern data engineering tools for scalable ingestion, processing, transformation, and analytics of continuous data streams.","language":"Python","stars":24,"topics":["github","shell","consumers","docker","monitoring","processors","producers","python","data","data-engineering"],"license":null,"category":"data-pipelines","readme_excerpt":"Real-Time Streaming Pipeline A production-ready event streaming pipeline with IoT, clickstream, and M-Pesa ingestion, Avro schema enforcement, Apache Flink processing, and resilient sink architecture. Overview This repository demonstrates a complete streaming architecture with: - Avro-encoded producers for IoT, clickstream, and M-Pesa events - Schema Registry compatibility and evolution testing - Kafka topic catalog with dev/prod replication settings - PyFlink aggregation and anomaly detection - PostgreSQL and Redis sinks for persistent and real-time data - Dead-letter queue consumer for failed event handling - GitHub Actions CI for linting, type checking, and tests Architecture Key Engineering Decisions - Flink over Spark Streaming : true stateful per-key processing with low latency and no micro-batching overhead. - Avro + Schema Registry : strict schema enforcement, compact binary payloads, and safe schema evolution. - Redis for real-time state : Redis is used for sub-second dashboards while PostgreSQL stores durable historical aggregates. - Z-score threshold rationale : the default 3.0 keeps false positives low for normally-distributed metrics and remains configurable. - DLQ isolation : failed events are routed to a dedicated dead-letter topic and consumer, preventing pipeline blockage. What’s Included - requirements.txt and requirements-dev.txt - pyproject.toml with mypy , black , and pytest settings - config/topics.yaml for Kafka topic definitions - scripts/create topics","default_branch":null,"files":null,"tree":[],"storefront":"/r/Victor-Kipruto-Rop","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Victor-Kipruto-Rop/streaming-pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}