{"repo":"guidok91/spark-structured-streaming-kafka","free":true,"listed":false,"github":"https://github.com/guidok91/spark-structured-streaming-kafka","clone":"git clone https://github.com/guidok91/spark-structured-streaming-kafka.git","description":"Spark Structured Streaming data pipeline that processes movie ratings data in real-time.","language":"Python","stars":14,"topics":["spark","etl","pyspark","real-time","data-engineering","spark-structured-streaming","kafka","apache-spark","apache-kafka","streaming"],"license":null,"category":"data-pipelines","readme_excerpt":"Spark Structured Streaming Demo Spark Structured Streaming data pipeline that processes movie ratings data in real-time. Consumes events from a Kafka topic in Avro, transforms and writes to an Apache Iceberg table. The pipeline handles updates and duplicate events by merging to the destination table based on the event id . The output table is partitioned by days(rating timestamp) (leveraging Iceberg's hidden partitioning for optimal querying) - Data Architecture - Local setup - Dependency management - Running instructions - Table internal maintenance Data Architecture Local setup We spin up a local Kafka cluster with Schema Registry based on the Docker Compose file provided by Confluent. We install a local Spark Structured Streaming app using uv. Dependency management Dependabot is configured to periodically upgrade repo dependencies. See dependabot.yml. Running instructions Run the following commands in order: make setup to install the Spark Structured Streaming app on a local Python env. make kafka-up to start local Kafka in Docker. make kafka-create-topic to create the Kafka topic we will use. make kafka-produce-test-events to start writing messages to the topic. On a separate console, run: make streaming-app-run to start the Spark Structured Streaming app. On a separate console, you can check the output dataset by running: Table internal maintenance The streaming microbatches can produce many small files and constant table snapshots. In order to tackle these issues, the r","default_branch":null,"files":null,"tree":[],"storefront":"/r/guidok91","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/guidok91/spark-structured-streaming-kafka/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}