{"repo":"eschizoid/kpipe","free":true,"listed":false,"github":"https://github.com/eschizoid/kpipe","clone":"git clone https://github.com/eschizoid/kpipe.git","description":"Composable Kafka consumer library for building modular, testable JVM data pipelines.","language":"Java","stars":57,"topics":["apache-kafka","data-pipelines","event-driven","functional-programming","java","kafka","stream-processing"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"kpipe KPipe is a plain-JVM Kafka consumer runtime for Java 25+: one virtual thread per record, a typed pipeline API, and at-least-once delivery that still holds while records run in parallel. KPipe replaces the hand-rolled Consumer.poll loop that most Kafka services grow around their business logic. You declare a pipeline — deserialize, transform, sink — and KPipe runs it with: - A virtual thread per record. I/O-bound work (DB writes, HTTP calls) overlaps without a worker pool to configure. You still bound in-flight work — records currently being processed — which KPipe's backpressure does by pausing consumption past a watermark; and your downstream resources — connection pools and rate limits — don't disappear because threads got cheap. - At-least-once delivery, held under parallelism. An offset is committed only after its record reaches a terminal state (sink completed, filtered, or parked in the dead-letter topic, the DLQ), and commits never pass a record that is still in flight. The exact guarantee boundary and failure matrix: docs/GUARANTEES.md. - One deserialize, one serialize per processing attempt. Operators transform the typed payload ( Map , GenericRecord , Message ) between a single decode and at most one encode (typed sinks skip it), instead of re-serializing between steps. - Typed outcomes instead of nulls. Every record ends as Passed , Filtered , or Failed — a sealed type, so \"intentionally skipped\" and \"broken\" cannot be confused, in code or in metrics. - The o","default_branch":null,"files":null,"tree":[],"storefront":"/r/eschizoid","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/eschizoid/kpipe/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}