{"repo":"AbsaOSS/hyperdrive","free":true,"listed":false,"github":"https://github.com/AbsaOSS/hyperdrive","clone":"git clone https://github.com/AbsaOSS/hyperdrive.git","description":"Extensible streaming ingestion pipeline on top of Apache Spark","language":"Scala","stars":47,"topics":["ingestion","spark-structured-streaming","spark","kafka","streaming","pipeline","framework","apache-spark","streaming-etl"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Copyright 2018 ABSA Group Limited Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Hyperdrive - An extensible streaming ingestion pipeline on top of Apache Spark Build Status master develop ------------- ------------- What is Hyperdrive? Hyperdrive is a configurable and scalable ingestion platform that allows data movement and transformation from streaming sources with exactly-once fault-tolerance semantics by using Apache Spark Structured Streaming. In Hyperdrive, each ingestion is defined by the three components reader, transformer and writer. This separation allows adapting to different streaming sources and sinks, while reusing transformations common across multiple ingestion pipelines. Motivation Similar to batch processing, data ingestion pipelines are needed to process streaming data sources. While solutions for data pipelines exist, exactly-once fault-tolerance in streaming processing is an intricate problem and cannot be solved with the same strategies that exist for batch processing. This is the gap the Hy","default_branch":null,"files":null,"tree":[],"storefront":"/r/AbsaOSS","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/AbsaOSS/hyperdrive/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}