{"repo":"streamnative/pulsar-spark","free":true,"listed":false,"github":"https://github.com/streamnative/pulsar-spark","clone":"git clone https://github.com/streamnative/pulsar-spark.git","description":"Spark Connector to read and write with Pulsar","language":"Scala","stars":120,"topics":["apache-pulsar","apache-spark","spark","structured-streaming","spark-sql","data-processing","data-science","batch-processing","stream-processing","flink"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"pulsar-spark Unified data processing with Apache Pulsar and Apache Spark. Prerequisites - Java 17 or later - Spark 4.1.1 or later - Pulsar 3.0 or later Version Compatibility Matrix The following table shows the tested and supported version combinations: Connector Version Spark Version Pulsar Client Pulsar Service Scala Version Status ------------------- --------------- --------------- ---------------- --------------- -------- 4.1.1.x 4.1.1 4.0.5 3.0 - 4.x 2.13.17 Current 4.0.1.x 4.0.1 4.0.5 3.0 - 4.x 2.13.16 Stable 3.5.6.x 3.5.6 4.0.5 3.0 - 4.x 2.13 Stable 3.5.2.x 3.5.2 4.0.5 3.0 - 4.x 2.13 Stable 3.4.1.x 3.4.1 2.10.5 2.10 - 3.x 2.13 Legacy 3.4.0.x 3.4.0 2.10.2 2.10 - 3.x 2.13 Legacy Notes : - Connector Version : Follows the Spark major.minor version pattern - Pulsar Client : The version of Pulsar client library bundled with the connector - Pulsar Service : Compatible Pulsar broker/service versions - Each connector version is built and tested against the specific Spark version listed - Pulsar client 4.x provides the best compatibility with Pulsar service 3.0+ clusters - For Pulsar 2.x clusters, use connector versions 3.4.x with Pulsar client 2.x Preparations Link Client library For Scala/Java applications using SBT/Maven project definitions, link your application with the following artifact: Deploy Client library As with any Spark applications, spark-submit is used to launch your application. pulsar-spark-connector {{SCALA BINARY VERSION}} and its dependencies can be directly","default_branch":null,"files":null,"tree":[],"storefront":"/r/streamnative","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/streamnative/pulsar-spark/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}