{"repo":"GoogleCloudPlatform/cloud-composer-mssql-dataflow-bigquery","free":true,"listed":false,"github":"https://github.com/GoogleCloudPlatform/cloud-composer-mssql-dataflow-bigquery","clone":"git clone https://github.com/GoogleCloudPlatform/cloud-composer-mssql-dataflow-bigquery.git","description":"This repository contains an example of how to leverage Cloud Composer and Cloud Dataflow to move data from a Microsoft SQL Server to BigQuery. The diagrams below demonstrate the workflow pipeline.","language":"Python","stars":20,"topics":["cloud-composer","airflow","bigquery","dataflow","microsoft-sql-server"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Cloud Composer Orchestrating Moving Data from Microsoft SQL Server to BigQuery Google Cloud Composer Example This repository contains an example of how to leverage Cloud Composer and Cloud Dataflow to move data from a Microsoft SQL Server to BigQuery. The diagrams below demonstrate the workflow pipeline. The Pipeline Steps are as follows: 1. A Cloud Composer DAG is either scheduled or manually triggered which connects a Microsoft SQL Server defined and exports the defined data to Google Cloud Storage as a JSON file. 2. A second Cloud Composer DAG is triggered by a Cloud Function once the JSON file has been written to the storage bucket. 3. The second Cloud Composer DAG triggers a Dataflow batch job which can if needed perform transformations then it writes the data to BigQuery. 5. Included in both Cloud Composer DAGs is the ability to send email notifications. You can: Schedule the Cloud Composer DAG to export data as needed with date filters. Perform transformation in Dataflow. Get a notification on a successful or failed jobs. Requirements: You need a Microsoft SQL Server installed either in Google Cloud or elsewhere. How to install 1. Install the Google Cloud SDK 2. Create a export storage bucket for Microsoft SQL Server Exports 3. Create a Dataflow staging storage bucket 4. Through the Google Cloud Console create a folder named tmp in the newly created bucket for the DataFlow staging files 5. Create a Cloud Composer Environment You need to use an image equal to or greater","default_branch":null,"files":null,"tree":[],"storefront":"/r/GoogleCloudPlatform","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/GoogleCloudPlatform/cloud-composer-mssql-dataflow-bigquery/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}