{"repo":"Aiven-Open/tiered-storage-for-apache-kafka","free":true,"listed":false,"github":"https://github.com/Aiven-Open/tiered-storage-for-apache-kafka","clone":"git clone https://github.com/Aiven-Open/tiered-storage-for-apache-kafka.git","description":"RemoteStorageManager for Apache Kafka® Tiered Storage","language":"Java","stars":238,"topics":["apache-kafka","kafka","tiered-storage"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Tiered Storage for Apache Kafka® This project is an implementation of RemoteStorageManager for Apache Kafka tiered storage. The implementation will have multiple configurable storage backends. Currently, AWS S3, Google Cloud Storage, and Azure Blob Storage are supported. The project follows the API specifications according to the latest version of KIP-405: Kafka Tiered Storage. NEW! Iceberg mode 🧊🎉 The Iceberg mode was added recently. It's currenly in the \"alpha\" stage, but could already be explored. Please have a look at the whitepaper and demo. References - Configuration - Metrics Getting started You can see the demo/ directory for some configured setups. Here are some steps you need to make to your brokers to get started with the plugin. Step 1. Prepare the plugin : The plugin consists of the core library ( core- .tgz ) and object storage-specific libraries ( azure- .tgz , gcs- .tgz , s3- .tgz ). Extract the core library and selected storage library to the same or different directories. Step 2. Configure the brokers : [!NOTE] For Remote Log Metadata Manager (RLMM) and further Tiered Storage configurations, see https://kafka.apache.org/documentation/#tieredstorageconfigs You may want to tweak remote.log.manager.task.interval.ms and log.retention.check.interval.ms to see the tiered storage effects faster. However, you probably don't need to change this in production setups. Step 3. Start the broker . Step 4. Create and fill the topic . Step 5 . After some time, you will st","default_branch":null,"files":null,"tree":[],"storefront":"/r/Aiven-Open","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Aiven-Open/tiered-storage-for-apache-kafka/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}