{"repo":"bakdata/kafka-large-message-serde","free":true,"listed":false,"github":"https://github.com/bakdata/kafka-large-message-serde","clone":"git clone https://github.com/bakdata/kafka-large-message-serde.git","description":"A Kafka Serde that reads and writes records from and to Blob storage (S3, Azure, Google) transparently.","language":"Java","stars":67,"topics":["azure-blob-storage","deserialization","google-cloud-storage","kafka","kafka-streams","large-data","s3","serde","serialization","simple-storage-service"],"license":"MIT","category":"data-pipelines","readme_excerpt":"kafka-large-message-serde A Kafka Serde that reads and writes records from and to a blob storage, such as Amazon S3, Azure Blob Storage, and Google Cloud Storage, transparently. Formerly known as kafka-s3-backed-serde. Getting Started Serde You can add kafka-large-message-serde via Maven Central. Gradle Maven For other build tools or versions, refer to the latest version in MvnRepository. Usage You can use it from your Kafka Streams application like any other Serde The following configuration options are available: large.message.key.serde Key serde class to use. All serde configurations are also delegated to this serde. Type: class Default: org.apache.kafka.common.serialization.Serdes$ByteArraySerde Importance: high large.message.value.serde Value serde class to use. All serde configurations are also delegated to this serde. Type: class Default: org.apache.kafka.common.serialization.Serdes$ByteArraySerde Importance: high large.message.base.path Base path to store data. Must include bucket and any prefix that should be used, e.g., s3://my-bucket/my/prefix/ . Available protocols: s3 , abs . Type: string Default: \"\" Importance: high large.message.max.byte.size Maximum serialized message size in bytes before messages are stored on blob storage. Type: int Default: 1000000 Importance: medium large.message.use.headers Enable if Kafka message headers should be used to distinguish blob storage backed messages. This is disabled by default for backwards compatibility but leads to increa","default_branch":null,"files":null,"tree":[],"storefront":"/r/bakdata","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/bakdata/kafka-large-message-serde/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}