{"repo":"SergeySenigov/data-engineer-practicum-portfolio","free":true,"listed":false,"github":"https://github.com/SergeySenigov/data-engineer-practicum-portfolio","clone":"git clone https://github.com/SergeySenigov/data-engineer-practicum-portfolio.git","description":"Проекты курса Инженер данных на платформе Yandex Practicum","language":"Python","stars":10,"topics":["airflow","docker","kafka","mongodb","python","spark","spark-streaming","sql","vertica","bash"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Data Engineer Course Projects Project Stack, tools, libraries :-------------------- :--------------------- Data quality cheks. RFM datamart. SQL, Common Table Expression, Window Functions, PostgreSQL, cloudbeaver Modifying DWH. Migration to the new model. SQL, Window Functions, PostgreSQL, cloudbeaver Modifying ETL and datamarts. Implementing idempotency. AirFlow, SQL, PostgreSQL, cloudbeaver, bash, pandas, SQLAlchemy, PostgresOperator, BashOperator Data quality checks in ETL AirFlow, SQL, PostgreSQL Datamart in DWH based on multiple sources Airflow, PostgreSQL, MongoDB Compass, pendulum, Jupyter Notebook, bash, SQLAlchemy, PostgresHook Datamart based on Analytical Database Vertica AirFlow, Yandex S3 Storage, Common Table Expression, SQL, Vertica, cloudbeaver, pandas Working with PySpark in Hadoop. Working with HDFS. Hadoop, Spark, PySpark, YARN, MapReduce, Window Functions, HDFS, Airflow, SparkSubmitOperator, Parquet Processing stream data with Spark Kafka, PySpark, AirFlow, kcat, Jupyter Notebook, SQL, PostgreSQL, Spark Streaming Cloud services Yandex Cloud Services, Datalense, Kubernetes, kubectl, Kafka, kcat, confluent kafka, flask, Docker Compose, Helm, Redis Combining data streams. Analytics datamart. Yandex S3, DWH, Vertica, boto3, Airflow, TriggerDagRunOperator, Metabase ---","default_branch":null,"files":null,"tree":[],"storefront":"/r/SergeySenigov","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/SergeySenigov/data-engineer-practicum-portfolio/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}