{"repo":"qwshen/spark-flight-connector","free":true,"listed":false,"github":"https://github.com/qwshen/spark-flight-connector","clone":"git clone https://github.com/qwshen/spark-flight-connector.git","description":"A Spark Connector that reads data from / writes data to Arrow-Flight end-points with Arrow-Flight and Flight-SQL","language":"Java","stars":49,"topics":["apache-arrow","apache-spark","arrow","arrow-flight","dremio","sql","apache-flight","data-source-api","flight-sql","spark-connector"],"license":"Apache-2.0","category":"databases-storage","readme_excerpt":"spark-flight-connector is an Apache Spark DataSource API that reads/writes data from/to arrow-flight endpoints, such as Dremio Flight Server. With proper partitioning, it supports fast loading of large datasets by parallelizing reads. It also supports all insert/merge/update/delete DML operations for writes. With arrow-flight, it enables high speed data transfers compared to ODBC/JDBC connections by utilizing the Apache Arrow format to avoid serializing and deserializing data. To build the project, run: The connector requires Spark 3.2.0+ due to the support of Interval types. For a quick start, please jump to this tutorial. In order to connect to an arrow-flight endpoint, the following properties are mandatory: - host : the full host-name or ip of an arrow-flight server; - port : the port number; - user : the user account for connecting to and reading/writing data from/to the arrow-flight endpoint; - password or bearerToken : the password/pat or auth2 token of the user account. One of them must be provided; the password takes precedence if both provided. - table : the name of a table from/to which the connector reads/writes data. The table can be a physical data table, or any view. It can also be a select sql-statement or tables-joining statement. For example: - Select statement: - Join tables statement: Note: the connector doesn't support legacy flight authentication mode (flight.auth.mode = legacy.arrow.flight.auth). The following properties are optional: - tls.enabled : wh","default_branch":null,"files":null,"tree":[],"storefront":"/r/qwshen","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/qwshen/spark-flight-connector/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}