{"repo":"yesdeepakverma/pysql-beam","free":true,"listed":false,"github":"https://github.com/yesdeepakverma/pysql-beam","clone":"git clone https://github.com/yesdeepakverma/pysql-beam.git","description":null,"language":"Python","stars":25,"topics":["python","apache-beam","gcp","dataflow","mysql","postgresql","mssql"],"license":"MIT","category":"databases-storage","readme_excerpt":"pysql-beam NOTE: We are looking for maintainer for this repository. We will really appreciate contribution. This package aim to provide Apache beam io connector for MySQL, Postgres and MSSQL Database database. This package provides apache beam io connectors for postgres db, mssql db and mysql db. This package is a python implementation for those 3 io connectors FYI: it uses a pyodbc connector for the mssql implementation, but not for the other two connectors Requirements: 1. Python =2.7 or python = 3.5 2. Apache beam = 2.10 3. pymysql[rsa] 4. psycopg2-binary 5. pyodbc Installation: 1. pip install pysql-beam or 1. pip install git+git@github.com:yesdeepakverma/pysql-beam.git Current functionality: 1. Read from MySQL database by passing either table name or sql query 2. Read from Postgres database by passing either table name or sql query 3. Read from MSSQL database by passing either table name or squl query 4. Write to BigQuery Reference Guide: 1. Java IO connector for the same: https://github.com/spotify/dbeam 2. How to write io connector for Apache Beam: https://beam.apache.org/documentation/io/developing-io-overview/ https://beam.apache.org/documentation/io/developing-io-python/ Usage Guide: Examples: For mysql: python cloud sql to bigquery.py --host localhost --port 3306 --database SECRET DATABASE --username SECRET USER --password SECRET PASSWORD --table YOUR TABLE --output table 'MyProject:MyDataset.MyTable' --temp location \"gs://MyBucket/tmp\" For postgres: python cloud sq","default_branch":null,"files":null,"tree":[],"storefront":"/r/yesdeepakverma","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/yesdeepakverma/pysql-beam/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}