{"repo":"pinterest/tiered-storage","free":true,"listed":false,"github":"https://github.com/pinterest/tiered-storage","clone":"git clone https://github.com/pinterest/tiered-storage.git","description":"Pinterest's simplified and efficient Tiered Storage implementation for Kafka","language":"Java","stars":26,"topics":["kafka","pubsub","tiered-storage"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Pinterest Tiered Storage for Apache Kafka® Pinterest Tiered Storage for Apache Kafka® is a broker-independent framework that allows brokers to offload finalized log segments to a remote storage system. This allows Apache Kafka® to maintain a smaller disk footprint and reduce the need for expensive storage on the brokers. The framework also provides a consumer client that can consume from both the broker and the remote storage system. Pinterest's implementation of Tiered Storage for Apache Kafka® provides a broker-independent approach to Tiered Storage. See the differences between Pinterest vs. Apache Kafka® Tiered Storage . It consists of two main components: 1. Uploader: A continuous process that runs on each Apache Kafka® broker and uploads finalized log segments to a remote storage system (e.g. Amazon S3, with unique prefix per cluster and topic). 2. Consumer: A consumer client capable of consuming from both Tiered Storage log segments and Apache Kafka® cluster. A third module ts-common contains common classes and interfaces that are used by the ts-consumer and ts-segment-uploader modules, such as Metrics, StorageEndpointProvider, etc. Feel free to read into each module's README for more details. Why Tiered Storage? Apache Kafka® is a distributed event streaming platform that stores partitioned and replicated log segments on disk for a configurable retention period. However, as data volume and/or retention periods grow, the disk footprint of Apache Kafka® clusters can beco","default_branch":null,"files":null,"tree":[],"storefront":"/r/pinterest","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/pinterest/tiered-storage/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}