{"repo":"aws-samples/iceberg-streaming-examples","free":true,"listed":false,"github":"https://github.com/aws-samples/iceberg-streaming-examples","clone":"git clone https://github.com/aws-samples/iceberg-streaming-examples.git","description":"This repo contains examples of high throughput ingestion using Apache Spark and Apache Iceberg. These examples cover IoT and CDC scenarios using best practices. The code can be deployed into any Spark compatible engine like Amazon EMR Serverless or AWS Glue. A fully local developer environment is also provided.","language":"Java","stars":30,"topics":["apache-iceberg","apache-spark","structured-streaming"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Streaming Apache Iceberg examples using Apache Spark Apache Kafka has become the de facto standard for building real-time data pipelines, but ingesting and storing large amounts of streaming data in a scalable and performant way is a complex, resource-intensive task. This project shows how an open table format - Apache Iceberg - combined with Apache Kafka and Apache Spark Structured Streaming addresses those challenges with high-throughput streaming ingestion. The focus here goes further than the typical PoC that consumes a few messages or small CSV files: the examples aim to sustain around 400,000 msg/sec in every scenario. Two example domains are used throughout: IoT: EV vehicle telemetry - a simple, flat VehicleTelemetry reading (vehicle id, event time, model, speed, state of charge, odometer, charging flag), produced at high rate in Protocol Buffers, Avro or JSON , ingested into Iceberg with several write strategies. CDC: bank-account changes - a DMS-like change feed (insert/update/delete with a monotonic source sequence) mirrored into an Iceberg table with a guarded MERGE. The concepts are applicable to PySpark or Scala programs with little effort - we program the transformations, which are converted to a logical plan and executed by the JVM (or by native engines such as Apache DataFusion Comet, Velox or Photon). Why Java? Easy library reuse, performant UDFs, and a friendly local development environment where everything can be debugged with breakpoints. A full PySpark co","default_branch":null,"files":null,"tree":[],"storefront":"/r/aws-samples","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/aws-samples/iceberg-streaming-examples/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}