{"repo":"aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue","free":true,"listed":false,"github":"https://github.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue","clone":"git clone https://github.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue.git","description":"Stream CDC into an Amazon S3 data lake in Apache Iceberg table format with AWS Glue Streaming and DMS","language":"Python","stars":36,"topics":["apache-iceberg","aws-glue","aws-dms","aws-athena","apache-spark"],"license":"MIT-0","category":"data-pipelines","readme_excerpt":"Transactional Data Lake using Apache Iceberg with AWS Glue Streaming and DMS This repository provides you cdk scripts and sample code on how to implement end to end data pipeline for transactional data lake by ingesting stream change data capture (CDC) from MySQL DB to Amazon S3 in Apache Iceberg format through Amazon Kinesis using Amazon Data Migration Service(DMS) and Glue Streaming. Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming and DMS Below diagram shows what we are implementing. The cdk.json file tells the CDK Toolkit how to execute your app. This project is set up like a standard Python project. The initialization process also creates a virtualenv within this project, stored under the .venv directory. To create the virtualenv it assumes that there is a python3 (or python for Windows) executable in your path with access to the venv package. If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv manually. To manually create a virtualenv on MacOS and Linux: After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv. If you are a Windows platform, you would activate the virtualenv like this: Once the virtualenv is activated, you can install the required dependencies. To add additional dependencies, for example other CDK libraries, just add them to your setup.py file and rerun the pip install -r requirements.txt command. Prerequis","default_branch":null,"files":null,"tree":[],"storefront":"/r/aws-samples","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}