{"repo":"parvvaresh/Data-Pipelines-for-Real-Time-Monitoring","free":true,"listed":false,"github":"https://github.com/parvvaresh/Data-Pipelines-for-Real-Time-Monitoring","clone":"git clone https://github.com/parvvaresh/Data-Pipelines-for-Real-Time-Monitoring.git","description":"This repository contains a small, self-contained example platform that demonstrates a real-time data pipeline using Kafka (KRaft mode), Logstash, Elasticsearch and Kibana","language":"Go","stars":12,"topics":["elasticsearch","kafka"],"license":null,"category":"data-pipelines","readme_excerpt":"Data Pipelines for Real-Time Monitoring This repository contains a small, self-contained example platform that demonstrates a real-time data pipeline using Kafka (KRaft mode), Logstash, Elasticsearch and Kibana. A lightweight synthetic producer (the fake-app ) generates JSON log events and pushes them to Kafka. Logstash consumes those messages, performs basic parsing, and indexes them into Elasticsearch for analysis and visualization in Kibana. The setup is optimized for local development and testing using Docker Compose. Architecture and runtime flow - Kafka (Bitnami image) runs in KRaft mode (no Zookeeper). It exposes PLAINTEXT listener on port 9092. - kafka-init is a one-time helper container that waits for Kafka to be ready and creates the configured topic(s). - fake-app is a Go-based producer that emits synthetic log events to the configured Kafka topic at a configurable rate. - kafka-ui (Provectus Kafka UI) provides a web interface for inspecting topics and messages. - Logstash subscribes to the Kafka topic, applies light parsing and field normalisation, and writes events to Elasticsearch. - Elasticsearch stores the indexed logs; Kibana connects to Elasticsearch to provide a dashboard and discovery UI. Service interactions (high level): 1. Start Docker Compose. Services start in the order defined in docker-compose.yml and controlled by depends on conditions and healthchecks. 2. Kafka accepts messages from producers at kafka:9092 . 3. kafka-init waits until Kafka is heal","default_branch":null,"files":null,"tree":[],"storefront":"/r/parvvaresh","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/parvvaresh/Data-Pipelines-for-Real-Time-Monitoring/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}