{"repo":"ConduitIO/conduit-connector-kafka","free":true,"listed":false,"github":"https://github.com/ConduitIO/conduit-connector-kafka","clone":"git clone https://github.com/ConduitIO/conduit-connector-kafka.git","description":"Conduit connector for Apache Kafka","language":"Go","stars":11,"topics":["kafka","go","golang","conduit"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Conduit Connector Kafka The Kafka connector is one of Conduit builtin plugins. It provides both, a source and a destination connector for Apache Kafka. Source A Kafka source connector is represented by a single consumer in a Kafka consumer group. By virtue of that, a source's logical position is the respective consumer's offset in Kafka. Internally, though, we're not saving the offset as the position: instead, we're saving the consumer group ID, since that's all which is needed for Kafka to find the offsets for our consumer. A source is getting associated with a consumer group ID the first time the Read() method is called. Destination The destination connector writes records to Kafka. Output format The output format can be adjusted using configuration options provided by the connector SDK: - sdk.record.format : used to choose the format - sdk.record.format.options : used to configure the specifics of the chosen format See this article for more info on configuring the output format. Batching Batching can also be configured using connector SDK provided options: - sdk.batch.size : maximum number of records in batch before it gets written to the destination (defaults to 0, no batching) - sdk.batch.delay : maximum delay before an incomplete batch is written to the destination (defaults to 0, no limit) Source Configuration Parameters Destination Configuration Parameters How to build? Run make build to build the connector. Testing Run make test to run all the unit and integration te","default_branch":null,"files":null,"tree":[],"storefront":"/r/ConduitIO","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ConduitIO/conduit-connector-kafka/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}