{"repo":"adform/stream-loader","free":true,"listed":false,"github":"https://github.com/adform/stream-loader","clone":"git clone https://github.com/adform/stream-loader.git","description":"Components for building stream loaders from Kafka to arbitrary storages","language":"Scala","stars":41,"topics":["kafka","vertica","clickhouse","s3","hdfs","iceberg"],"license":"MPL-2.0","category":"data-pipelines","readme_excerpt":"Stream Loader Stream loader is a collection of libraries providing means to load data from Kafka into arbitrary storages such as Iceberg, HDFS, S3, ClickHouse or Vertica using exactly-once semantics. Users can easily implement various highly customized loaders by combining out of the box components for record formatting and encoding, data storage, stream grouping and so on, implementing their own or extending the existing ones, if needed. Getting Started Stream loader is currently available for Scala 2.13 only, to get started add the following to your build.sbt : Various storage specific parts are split into separate artifacts to reduce dependencies and coupling, currently the following artifacts are available: - stream-loader-core : core functionality (required) . - stream-loader-clickhouse : ClickHouse storage and binary file format encoders. - stream-loader-hadoop : HDFS storage, components for working with parquet files. - stream-loader-iceberg : Iceberg table sink. - stream-loader-s3 : S3 compatible storage. - stream-loader-vertica : Vertica storage and native file format encoders. As an example we will be implementing an S3 loader, hence we will additionally require the stream-loader-s3 dependency. Any loader is an ordinary JVM application that simply runs an instance of a StreamLoader (or multiple instances in separate threads if higher throughput is needed): All loaders require a source and a sink , where the source is always Kafka: and the sink can be arbitrary, in c","default_branch":null,"files":null,"tree":[],"storefront":"/r/adform","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/adform/stream-loader/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}